2023-04-19 07:53:33,397 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0 2023-04-19 07:53:33,413 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-19 07:53:33,432 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-19 07:53:33,432 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/cluster_d18a6997-17eb-f6df-6ce9-376c883e1a31, deleteOnExit=true 2023-04-19 07:53:33,432 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-19 07:53:33,433 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/test.cache.data in system properties and HBase conf 2023-04-19 07:53:33,433 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/hadoop.tmp.dir in system properties and HBase conf 2023-04-19 07:53:33,434 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/hadoop.log.dir in system properties and HBase conf 2023-04-19 07:53:33,434 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-19 07:53:33,434 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-19 07:53:33,434 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-19 07:53:33,559 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-19 07:53:33,997 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-19 07:53:34,003 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-19 07:53:34,003 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-19 07:53:34,003 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-19 07:53:34,004 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-19 07:53:34,004 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-19 07:53:34,004 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-19 07:53:34,005 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-19 07:53:34,005 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-19 07:53:34,006 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-19 07:53:34,006 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/nfs.dump.dir in system properties and HBase conf 2023-04-19 07:53:34,006 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/java.io.tmpdir in system properties and HBase conf 2023-04-19 07:53:34,007 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-19 07:53:34,007 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-19 07:53:34,007 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-19 07:53:34,589 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-19 07:53:34,594 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-19 07:53:34,903 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-19 07:53:35,089 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-19 07:53:35,111 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-19 07:53:35,150 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-19 07:53:35,183 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/java.io.tmpdir/Jetty_localhost_34637_hdfs____fson9f/webapp 2023-04-19 07:53:35,314 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34637 2023-04-19 07:53:35,353 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-19 07:53:35,353 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-19 07:53:35,794 WARN [Listener at localhost/36491] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-19 07:53:35,953 WARN [Listener at localhost/36491] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-19 07:53:35,978 WARN [Listener at localhost/36491] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-19 07:53:35,984 INFO [Listener at localhost/36491] log.Slf4jLog(67): jetty-6.1.26 2023-04-19 07:53:35,989 INFO [Listener at localhost/36491] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/java.io.tmpdir/Jetty_localhost_41177_datanode____85ywwr/webapp 2023-04-19 07:53:36,106 INFO [Listener at localhost/36491] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41177 2023-04-19 07:53:36,443 WARN [Listener at localhost/42517] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-19 07:53:37,006 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5782a7fb2036eaaf: Processing first storage report for DS-7f1681a9-284e-4939-af0e-7d69880199e3 from datanode a4cdd384-e065-4e1c-a4f7-c8d7ed63f27c 2023-04-19 07:53:37,008 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5782a7fb2036eaaf: from storage DS-7f1681a9-284e-4939-af0e-7d69880199e3 node DatanodeRegistration(127.0.0.1:41071, datanodeUuid=a4cdd384-e065-4e1c-a4f7-c8d7ed63f27c, infoPort=42001, infoSecurePort=0, ipcPort=42517, storageInfo=lv=-57;cid=testClusterID;nsid=753842757;c=1681890814670), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-19 07:53:37,008 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5782a7fb2036eaaf: Processing first storage report for DS-2bdcbe53-03ef-4d2b-97cc-a471ccdb7d33 from datanode a4cdd384-e065-4e1c-a4f7-c8d7ed63f27c 2023-04-19 07:53:37,008 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5782a7fb2036eaaf: from storage DS-2bdcbe53-03ef-4d2b-97cc-a471ccdb7d33 node DatanodeRegistration(127.0.0.1:41071, datanodeUuid=a4cdd384-e065-4e1c-a4f7-c8d7ed63f27c, infoPort=42001, infoSecurePort=0, ipcPort=42517, storageInfo=lv=-57;cid=testClusterID;nsid=753842757;c=1681890814670), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-19 07:53:37,060 DEBUG [Listener at localhost/42517] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0 2023-04-19 07:53:37,151 INFO [Listener at localhost/42517] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/cluster_d18a6997-17eb-f6df-6ce9-376c883e1a31/zookeeper_0, clientPort=58087, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/cluster_d18a6997-17eb-f6df-6ce9-376c883e1a31/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/cluster_d18a6997-17eb-f6df-6ce9-376c883e1a31/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-19 07:53:37,166 INFO [Listener at localhost/42517] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=58087 2023-04-19 07:53:37,173 INFO [Listener at localhost/42517] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-19 07:53:37,175 INFO [Listener at localhost/42517] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-19 07:53:37,850 INFO [Listener at localhost/42517] util.FSUtils(471): Created version file at hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29 with version=8 2023-04-19 07:53:37,851 INFO [Listener at localhost/42517] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/hbase-staging 2023-04-19 07:53:38,155 INFO [Listener at localhost/42517] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-19 07:53:38,618 INFO [Listener at localhost/42517] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-19 07:53:38,649 INFO [Listener at localhost/42517] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-19 07:53:38,650 INFO [Listener at localhost/42517] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-19 07:53:38,650 INFO [Listener at localhost/42517] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-19 07:53:38,650 INFO [Listener at localhost/42517] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-19 07:53:38,651 INFO [Listener at localhost/42517] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-19 07:53:38,809 INFO [Listener at localhost/42517] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-19 07:53:38,886 DEBUG [Listener at localhost/42517] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-19 07:53:38,995 INFO [Listener at localhost/42517] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43905 2023-04-19 07:53:39,006 INFO [Listener at localhost/42517] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-19 07:53:39,009 INFO [Listener at localhost/42517] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-19 07:53:39,032 INFO [Listener at localhost/42517] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43905 connecting to ZooKeeper ensemble=127.0.0.1:58087 2023-04-19 07:53:39,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:439050x0, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-19 07:53:39,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:43905-0x1019b691e330000 connected 2023-04-19 07:53:39,101 DEBUG [Listener at localhost/42517] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-19 07:53:39,102 DEBUG [Listener at localhost/42517] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-19 07:53:39,105 DEBUG [Listener at localhost/42517] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-19 07:53:39,113 DEBUG [Listener at localhost/42517] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43905 2023-04-19 07:53:39,113 DEBUG [Listener at localhost/42517] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43905 2023-04-19 07:53:39,114 DEBUG [Listener at localhost/42517] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43905 2023-04-19 07:53:39,114 DEBUG [Listener at localhost/42517] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43905 2023-04-19 07:53:39,114 DEBUG [Listener at localhost/42517] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43905 2023-04-19 07:53:39,121 INFO [Listener at localhost/42517] master.HMaster(444): hbase.rootdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29, hbase.cluster.distributed=false 2023-04-19 07:53:39,192 INFO [Listener at localhost/42517] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-19 07:53:39,192 INFO [Listener at localhost/42517] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-19 07:53:39,192 INFO [Listener at localhost/42517] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-19 07:53:39,193 INFO [Listener at localhost/42517] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-19 07:53:39,193 INFO [Listener at localhost/42517] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-19 07:53:39,193 INFO [Listener at localhost/42517] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-19 07:53:39,199 INFO [Listener at localhost/42517] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-19 07:53:39,202 INFO [Listener at localhost/42517] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42359 2023-04-19 07:53:39,205 INFO [Listener at localhost/42517] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-19 07:53:39,212 DEBUG [Listener at localhost/42517] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-19 07:53:39,213 INFO [Listener at localhost/42517] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-19 07:53:39,216 INFO [Listener at localhost/42517] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-19 07:53:39,218 INFO [Listener at localhost/42517] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42359 connecting to ZooKeeper ensemble=127.0.0.1:58087 2023-04-19 07:53:39,221 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:423590x0, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-19 07:53:39,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42359-0x1019b691e330001 connected 2023-04-19 07:53:39,222 DEBUG [Listener at localhost/42517] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-19 07:53:39,224 DEBUG [Listener at localhost/42517] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-19 07:53:39,225 DEBUG [Listener at localhost/42517] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-19 07:53:39,228 DEBUG [Listener at localhost/42517] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42359 2023-04-19 07:53:39,228 DEBUG [Listener at localhost/42517] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42359 2023-04-19 07:53:39,228 DEBUG [Listener at localhost/42517] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42359 2023-04-19 07:53:39,229 DEBUG [Listener at localhost/42517] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42359 2023-04-19 07:53:39,229 DEBUG [Listener at localhost/42517] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42359 2023-04-19 07:53:39,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,43905,1681890817995 2023-04-19 07:53:39,246 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-19 07:53:39,248 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,43905,1681890817995 2023-04-19 07:53:39,267 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-19 07:53:39,267 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-19 07:53:39,267 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 07:53:39,268 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-19 07:53:39,269 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,43905,1681890817995 from backup master directory 2023-04-19 07:53:39,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-19 07:53:39,272 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,43905,1681890817995 2023-04-19 07:53:39,273 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-19 07:53:39,273 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-19 07:53:39,273 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,43905,1681890817995 2023-04-19 07:53:39,276 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-19 07:53:39,277 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-19 07:53:39,368 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/hbase.id with ID: b322461b-5bb6-411b-85ee-c139b578b695 2023-04-19 07:53:39,410 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-19 07:53:39,425 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 07:53:39,467 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x162b1133 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:53:39,503 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bb453e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:53:39,533 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 07:53:39,535 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-19 07:53:39,555 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-19 07:53:39,555 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-19 07:53:39,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:53:39,561 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:53:39,563 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-19 07:53:39,602 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/MasterData/data/master/store-tmp 2023-04-19 07:53:40,033 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:53:40,033 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-19 07:53:40,033 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 07:53:40,033 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 07:53:40,034 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-19 07:53:40,034 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 07:53:40,034 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 07:53:40,034 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-19 07:53:40,036 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/MasterData/WALs/jenkins-hbase4.apache.org,43905,1681890817995 2023-04-19 07:53:40,056 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43905%2C1681890817995, suffix=, logDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/MasterData/WALs/jenkins-hbase4.apache.org,43905,1681890817995, archiveDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/MasterData/oldWALs, maxLogs=10 2023-04-19 07:53:40,114 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41071,DS-7f1681a9-284e-4939-af0e-7d69880199e3,DISK] 2023-04-19 07:53:40,123 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:53:40,191 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/MasterData/WALs/jenkins-hbase4.apache.org,43905,1681890817995/jenkins-hbase4.apache.org%2C43905%2C1681890817995.1681890820067 2023-04-19 07:53:40,192 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41071,DS-7f1681a9-284e-4939-af0e-7d69880199e3,DISK]] 2023-04-19 07:53:40,193 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-19 07:53:40,193 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:53:40,197 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-19 07:53:40,199 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-19 07:53:40,283 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-19 07:53:40,292 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-19 07:53:40,323 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-19 07:53:40,338 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:53:40,345 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-19 07:53:40,347 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-19 07:53:40,369 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-19 07:53:40,374 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 07:53:40,375 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59160187, jitterRate=-0.1184445172548294}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-19 07:53:40,375 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-19 07:53:40,377 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-19 07:53:40,404 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-19 07:53:40,404 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-19 07:53:40,407 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-19 07:53:40,409 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-19 07:53:40,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 44 msec 2023-04-19 07:53:40,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-19 07:53:40,486 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-19 07:53:40,493 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-19 07:53:40,524 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-19 07:53:40,529 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-19 07:53:40,532 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-19 07:53:40,539 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-19 07:53:40,543 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-19 07:53:40,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 07:53:40,548 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-19 07:53:40,548 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-19 07:53:40,562 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-19 07:53:40,566 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-19 07:53:40,566 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-19 07:53:40,566 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 07:53:40,567 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,43905,1681890817995, sessionid=0x1019b691e330000, setting cluster-up flag (Was=false) 2023-04-19 07:53:40,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 07:53:40,591 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-19 07:53:40,592 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43905,1681890817995 2023-04-19 07:53:40,601 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 07:53:40,612 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-19 07:53:40,614 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43905,1681890817995 2023-04-19 07:53:40,617 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.hbase-snapshot/.tmp 2023-04-19 07:53:40,634 INFO [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(952): ClusterId : b322461b-5bb6-411b-85ee-c139b578b695 2023-04-19 07:53:40,638 DEBUG [RS:0;jenkins-hbase4:42359] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-19 07:53:40,701 DEBUG [RS:0;jenkins-hbase4:42359] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-19 07:53:40,701 DEBUG [RS:0;jenkins-hbase4:42359] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-19 07:53:40,748 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-19 07:53:40,758 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-19 07:53:40,758 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-19 07:53:40,759 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-19 07:53:40,759 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-19 07:53:40,759 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-19 07:53:40,759 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 07:53:40,759 DEBUG [RS:0;jenkins-hbase4:42359] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-19 07:53:40,759 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-19 07:53:40,760 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 07:53:40,760 DEBUG [RS:0;jenkins-hbase4:42359] zookeeper.ReadOnlyZKClient(139): Connect 0x25146046 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:53:40,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681890850761 2023-04-19 07:53:40,764 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-19 07:53:40,771 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-19 07:53:40,772 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-19 07:53:40,776 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-19 07:53:40,778 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-19 07:53:40,784 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-19 07:53:40,785 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-19 07:53:40,786 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-19 07:53:40,786 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-19 07:53:40,787 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:40,788 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-19 07:53:40,790 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-19 07:53:40,791 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-19 07:53:40,794 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-19 07:53:40,794 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-19 07:53:40,796 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681890820795,5,FailOnTimeoutGroup] 2023-04-19 07:53:40,796 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681890820796,5,FailOnTimeoutGroup] 2023-04-19 07:53:40,796 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:40,796 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-19 07:53:40,798 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:40,798 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:40,811 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-19 07:53:40,813 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-19 07:53:40,813 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29 2023-04-19 07:53:40,825 DEBUG [RS:0;jenkins-hbase4:42359] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2bd39a94, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:53:40,826 DEBUG [RS:0;jenkins-hbase4:42359] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2fec0b36, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-19 07:53:40,830 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:53:40,833 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-19 07:53:40,836 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/info 2023-04-19 07:53:40,837 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-19 07:53:40,838 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:53:40,838 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-19 07:53:40,842 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/rep_barrier 2023-04-19 07:53:40,843 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-19 07:53:40,844 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:53:40,844 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-19 07:53:40,847 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/table 2023-04-19 07:53:40,847 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-19 07:53:40,848 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:53:40,850 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740 2023-04-19 07:53:40,852 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740 2023-04-19 07:53:40,855 DEBUG [RS:0;jenkins-hbase4:42359] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:42359 2023-04-19 07:53:40,856 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 07:53:40,858 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-19 07:53:40,860 INFO [RS:0;jenkins-hbase4:42359] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-19 07:53:40,860 INFO [RS:0;jenkins-hbase4:42359] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-19 07:53:40,860 DEBUG [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1023): About to register with Master. 2023-04-19 07:53:40,863 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 07:53:40,863 INFO [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,43905,1681890817995 with isa=jenkins-hbase4.apache.org/172.31.14.131:42359, startcode=1681890819191 2023-04-19 07:53:40,864 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68193134, jitterRate=0.016156882047653198}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 07:53:40,864 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-19 07:53:40,864 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-19 07:53:40,864 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-19 07:53:40,864 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-19 07:53:40,864 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-19 07:53:40,865 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-19 07:53:40,865 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-19 07:53:40,866 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-19 07:53:40,871 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-19 07:53:40,871 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-19 07:53:40,880 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-19 07:53:40,882 DEBUG [RS:0;jenkins-hbase4:42359] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-19 07:53:40,893 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-19 07:53:40,896 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-19 07:53:40,934 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51203, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-19 07:53:40,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:40,963 DEBUG [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29 2023-04-19 07:53:40,964 DEBUG [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36491 2023-04-19 07:53:40,964 DEBUG [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-19 07:53:40,969 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-19 07:53:40,970 DEBUG [RS:0;jenkins-hbase4:42359] zookeeper.ZKUtil(162): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:40,970 WARN [RS:0;jenkins-hbase4:42359] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-19 07:53:40,971 INFO [RS:0;jenkins-hbase4:42359] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-19 07:53:40,971 DEBUG [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/WALs/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:40,973 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42359,1681890819191] 2023-04-19 07:53:40,981 DEBUG [RS:0;jenkins-hbase4:42359] zookeeper.ZKUtil(162): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:40,993 DEBUG [RS:0;jenkins-hbase4:42359] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-19 07:53:41,002 INFO [RS:0;jenkins-hbase4:42359] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-19 07:53:41,025 INFO [RS:0;jenkins-hbase4:42359] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-19 07:53:41,030 INFO [RS:0;jenkins-hbase4:42359] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-19 07:53:41,030 INFO [RS:0;jenkins-hbase4:42359] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:41,031 INFO [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-19 07:53:41,040 INFO [RS:0;jenkins-hbase4:42359] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:41,040 DEBUG [RS:0;jenkins-hbase4:42359] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 07:53:41,041 DEBUG [RS:0;jenkins-hbase4:42359] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 07:53:41,042 DEBUG [RS:0;jenkins-hbase4:42359] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 07:53:41,042 DEBUG [RS:0;jenkins-hbase4:42359] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 07:53:41,043 DEBUG [RS:0;jenkins-hbase4:42359] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 07:53:41,043 DEBUG [RS:0;jenkins-hbase4:42359] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-19 07:53:41,043 DEBUG [RS:0;jenkins-hbase4:42359] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 07:53:41,043 DEBUG [RS:0;jenkins-hbase4:42359] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 07:53:41,043 DEBUG [RS:0;jenkins-hbase4:42359] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 07:53:41,043 DEBUG [RS:0;jenkins-hbase4:42359] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-19 07:53:41,044 INFO [RS:0;jenkins-hbase4:42359] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:41,045 INFO [RS:0;jenkins-hbase4:42359] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:41,045 INFO [RS:0;jenkins-hbase4:42359] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:41,048 DEBUG [jenkins-hbase4:43905] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-19 07:53:41,051 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42359,1681890819191, state=OPENING 2023-04-19 07:53:41,059 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-19 07:53:41,062 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 07:53:41,063 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-19 07:53:41,064 INFO [RS:0;jenkins-hbase4:42359] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-19 07:53:41,066 INFO [RS:0;jenkins-hbase4:42359] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42359,1681890819191-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:41,068 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:53:41,090 INFO [RS:0;jenkins-hbase4:42359] regionserver.Replication(203): jenkins-hbase4.apache.org,42359,1681890819191 started 2023-04-19 07:53:41,090 INFO [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,42359,1681890819191, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42359, sessionid=0x1019b691e330001 2023-04-19 07:53:41,090 DEBUG [RS:0;jenkins-hbase4:42359] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-19 07:53:41,090 DEBUG [RS:0;jenkins-hbase4:42359] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:41,090 DEBUG [RS:0;jenkins-hbase4:42359] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42359,1681890819191' 2023-04-19 07:53:41,090 DEBUG [RS:0;jenkins-hbase4:42359] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:41,091 DEBUG [RS:0;jenkins-hbase4:42359] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:41,092 DEBUG [RS:0;jenkins-hbase4:42359] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-19 07:53:41,092 DEBUG [RS:0;jenkins-hbase4:42359] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-19 07:53:41,092 DEBUG [RS:0;jenkins-hbase4:42359] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:41,092 DEBUG [RS:0;jenkins-hbase4:42359] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42359,1681890819191' 2023-04-19 07:53:41,092 DEBUG [RS:0;jenkins-hbase4:42359] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-19 07:53:41,093 DEBUG [RS:0;jenkins-hbase4:42359] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-19 07:53:41,093 DEBUG [RS:0;jenkins-hbase4:42359] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-19 07:53:41,093 INFO [RS:0;jenkins-hbase4:42359] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-19 07:53:41,094 INFO [RS:0;jenkins-hbase4:42359] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-19 07:53:41,206 INFO [RS:0;jenkins-hbase4:42359] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42359%2C1681890819191, suffix=, logDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/WALs/jenkins-hbase4.apache.org,42359,1681890819191, archiveDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/oldWALs, maxLogs=32 2023-04-19 07:53:41,228 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41071,DS-7f1681a9-284e-4939-af0e-7d69880199e3,DISK] 2023-04-19 07:53:41,233 INFO [RS:0;jenkins-hbase4:42359] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/WALs/jenkins-hbase4.apache.org,42359,1681890819191/jenkins-hbase4.apache.org%2C42359%2C1681890819191.1681890821210 2023-04-19 07:53:41,233 DEBUG [RS:0;jenkins-hbase4:42359] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41071,DS-7f1681a9-284e-4939-af0e-7d69880199e3,DISK]] 2023-04-19 07:53:41,271 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:41,273 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-19 07:53:41,277 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44526, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-19 07:53:41,289 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-19 07:53:41,289 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-19 07:53:41,294 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42359%2C1681890819191.meta, suffix=.meta, logDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/WALs/jenkins-hbase4.apache.org,42359,1681890819191, archiveDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/oldWALs, maxLogs=32 2023-04-19 07:53:41,317 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41071,DS-7f1681a9-284e-4939-af0e-7d69880199e3,DISK] 2023-04-19 07:53:41,329 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/WALs/jenkins-hbase4.apache.org,42359,1681890819191/jenkins-hbase4.apache.org%2C42359%2C1681890819191.meta.1681890821296.meta 2023-04-19 07:53:41,330 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41071,DS-7f1681a9-284e-4939-af0e-7d69880199e3,DISK]] 2023-04-19 07:53:41,330 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-19 07:53:41,332 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-19 07:53:41,349 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-19 07:53:41,356 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-19 07:53:41,365 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-19 07:53:41,365 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:53:41,366 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-19 07:53:41,366 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-19 07:53:41,372 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-19 07:53:41,374 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/info 2023-04-19 07:53:41,374 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/info 2023-04-19 07:53:41,375 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-19 07:53:41,380 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:53:41,380 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-19 07:53:41,382 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/rep_barrier 2023-04-19 07:53:41,382 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/rep_barrier 2023-04-19 07:53:41,384 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-19 07:53:41,385 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:53:41,386 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-19 07:53:41,388 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/table 2023-04-19 07:53:41,388 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/table 2023-04-19 07:53:41,388 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-19 07:53:41,390 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:53:41,392 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740 2023-04-19 07:53:41,396 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740 2023-04-19 07:53:41,401 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 07:53:41,404 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-19 07:53:41,405 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63807023, jitterRate=-0.04920126497745514}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 07:53:41,405 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-19 07:53:41,415 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681890821263 2023-04-19 07:53:41,433 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-19 07:53:41,434 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-19 07:53:41,435 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42359,1681890819191, state=OPEN 2023-04-19 07:53:41,437 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-19 07:53:41,437 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-19 07:53:41,442 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-19 07:53:41,442 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42359,1681890819191 in 370 msec 2023-04-19 07:53:41,447 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-19 07:53:41,447 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 564 msec 2023-04-19 07:53:41,451 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 782 msec 2023-04-19 07:53:41,452 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681890821452, completionTime=-1 2023-04-19 07:53:41,452 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-19 07:53:41,452 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-19 07:53:41,518 DEBUG [hconnection-0x52d79795-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:53:41,523 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44528, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:53:41,547 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-19 07:53:41,547 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681890881547 2023-04-19 07:53:41,547 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681890941547 2023-04-19 07:53:41,547 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 94 msec 2023-04-19 07:53:41,572 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43905,1681890817995-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:41,572 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43905,1681890817995-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:41,573 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43905,1681890817995-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:41,574 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:43905, period=300000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:41,575 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-19 07:53:41,583 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-19 07:53:41,595 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-19 07:53:41,596 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-19 07:53:41,607 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-19 07:53:41,610 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-19 07:53:41,614 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-19 07:53:41,641 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/hbase/namespace/6189702b8073b2778b13e3b2b656c082 2023-04-19 07:53:41,643 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/hbase/namespace/6189702b8073b2778b13e3b2b656c082 empty. 2023-04-19 07:53:41,644 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/hbase/namespace/6189702b8073b2778b13e3b2b656c082 2023-04-19 07:53:41,644 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-19 07:53:42,067 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-19 07:53:42,070 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6189702b8073b2778b13e3b2b656c082, NAME => 'hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp 2023-04-19 07:53:42,485 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:53:42,485 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 6189702b8073b2778b13e3b2b656c082, disabling compactions & flushes 2023-04-19 07:53:42,485 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082. 2023-04-19 07:53:42,485 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082. 2023-04-19 07:53:42,485 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082. after waiting 0 ms 2023-04-19 07:53:42,485 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082. 2023-04-19 07:53:42,485 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082. 2023-04-19 07:53:42,485 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 6189702b8073b2778b13e3b2b656c082: 2023-04-19 07:53:42,491 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-19 07:53:42,506 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681890822494"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890822494"}]},"ts":"1681890822494"} 2023-04-19 07:53:42,536 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-19 07:53:42,538 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-19 07:53:42,543 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890822538"}]},"ts":"1681890822538"} 2023-04-19 07:53:42,548 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-19 07:53:42,560 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=6189702b8073b2778b13e3b2b656c082, ASSIGN}] 2023-04-19 07:53:42,563 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=6189702b8073b2778b13e3b2b656c082, ASSIGN 2023-04-19 07:53:42,565 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=6189702b8073b2778b13e3b2b656c082, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42359,1681890819191; forceNewPlan=false, retain=false 2023-04-19 07:53:42,717 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=6189702b8073b2778b13e3b2b656c082, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:42,717 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681890822716"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890822716"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890822716"}]},"ts":"1681890822716"} 2023-04-19 07:53:42,723 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 6189702b8073b2778b13e3b2b656c082, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:53:42,884 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082. 2023-04-19 07:53:42,885 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6189702b8073b2778b13e3b2b656c082, NAME => 'hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082.', STARTKEY => '', ENDKEY => ''} 2023-04-19 07:53:42,886 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 6189702b8073b2778b13e3b2b656c082 2023-04-19 07:53:42,886 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:53:42,886 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6189702b8073b2778b13e3b2b656c082 2023-04-19 07:53:42,887 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6189702b8073b2778b13e3b2b656c082 2023-04-19 07:53:42,889 INFO [StoreOpener-6189702b8073b2778b13e3b2b656c082-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 6189702b8073b2778b13e3b2b656c082 2023-04-19 07:53:42,892 DEBUG [StoreOpener-6189702b8073b2778b13e3b2b656c082-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/namespace/6189702b8073b2778b13e3b2b656c082/info 2023-04-19 07:53:42,892 DEBUG [StoreOpener-6189702b8073b2778b13e3b2b656c082-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/namespace/6189702b8073b2778b13e3b2b656c082/info 2023-04-19 07:53:42,893 INFO [StoreOpener-6189702b8073b2778b13e3b2b656c082-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6189702b8073b2778b13e3b2b656c082 columnFamilyName info 2023-04-19 07:53:42,894 INFO [StoreOpener-6189702b8073b2778b13e3b2b656c082-1] regionserver.HStore(310): Store=6189702b8073b2778b13e3b2b656c082/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:53:42,896 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/namespace/6189702b8073b2778b13e3b2b656c082 2023-04-19 07:53:42,897 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/namespace/6189702b8073b2778b13e3b2b656c082 2023-04-19 07:53:42,903 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6189702b8073b2778b13e3b2b656c082 2023-04-19 07:53:42,908 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/namespace/6189702b8073b2778b13e3b2b656c082/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 07:53:42,909 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6189702b8073b2778b13e3b2b656c082; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74661058, jitterRate=0.1125364601612091}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-19 07:53:42,909 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6189702b8073b2778b13e3b2b656c082: 2023-04-19 07:53:42,911 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082., pid=6, masterSystemTime=1681890822877 2023-04-19 07:53:42,916 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082. 2023-04-19 07:53:42,916 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082. 2023-04-19 07:53:42,917 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=6189702b8073b2778b13e3b2b656c082, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:42,918 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681890822917"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681890822917"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681890822917"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681890822917"}]},"ts":"1681890822917"} 2023-04-19 07:53:42,925 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-19 07:53:42,925 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 6189702b8073b2778b13e3b2b656c082, server=jenkins-hbase4.apache.org,42359,1681890819191 in 198 msec 2023-04-19 07:53:42,929 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-19 07:53:42,929 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=6189702b8073b2778b13e3b2b656c082, ASSIGN in 365 msec 2023-04-19 07:53:42,930 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-19 07:53:42,931 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890822930"}]},"ts":"1681890822930"} 2023-04-19 07:53:42,933 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-19 07:53:42,937 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-19 07:53:42,940 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3400 sec 2023-04-19 07:53:43,011 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-19 07:53:43,014 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-19 07:53:43,014 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 07:53:43,053 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-19 07:53:43,073 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-19 07:53:43,079 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 35 msec 2023-04-19 07:53:43,092 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-19 07:53:43,104 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-19 07:53:43,109 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-04-19 07:53:43,119 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-19 07:53:43,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-19 07:53:43,124 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.850sec 2023-04-19 07:53:43,125 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-19 07:53:43,127 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-19 07:53:43,127 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-19 07:53:43,128 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43905,1681890817995-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-19 07:53:43,129 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43905,1681890817995-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-19 07:53:43,140 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-19 07:53:43,141 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x3ddd91d3 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:53:43,145 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51f71277, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:53:43,160 DEBUG [hconnection-0x286751c0-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:53:43,174 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44544, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:53:43,184 INFO [Listener at localhost/42517] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,43905,1681890817995 2023-04-19 07:53:43,205 INFO [Listener at localhost/42517] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=317, ProcessCount=169, AvailableMemoryMB=7126 2023-04-19 07:53:43,222 DEBUG [Listener at localhost/42517] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-19 07:53:43,225 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45756, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-19 07:53:43,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 07:53:43,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 07:53:43,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-19 07:53:43,247 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-19 07:53:43,250 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-19 07:53:43,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-19 07:53:43,254 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:43,256 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71 empty. 2023-04-19 07:53:43,258 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:43,258 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 07:53:43,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-19 07:53:43,279 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 07:53:43,281 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3c54cc0aa26fee1a2f14917923d1bb71, NAME => 'TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp 2023-04-19 07:53:43,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-19 07:53:43,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-19 07:53:43,703 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:53:43,704 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 3c54cc0aa26fee1a2f14917923d1bb71, disabling compactions & flushes 2023-04-19 07:53:43,704 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:43,704 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:43,704 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. after waiting 0 ms 2023-04-19 07:53:43,704 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:43,704 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:43,704 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:43,709 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-19 07:53:43,711 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890823710"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890823710"}]},"ts":"1681890823710"} 2023-04-19 07:53:43,714 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-19 07:53:43,715 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-19 07:53:43,716 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890823716"}]},"ts":"1681890823716"} 2023-04-19 07:53:43,719 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-19 07:53:43,727 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3c54cc0aa26fee1a2f14917923d1bb71, ASSIGN}] 2023-04-19 07:53:43,730 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3c54cc0aa26fee1a2f14917923d1bb71, ASSIGN 2023-04-19 07:53:43,732 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3c54cc0aa26fee1a2f14917923d1bb71, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42359,1681890819191; forceNewPlan=false, retain=false 2023-04-19 07:53:43,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-19 07:53:43,883 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=3c54cc0aa26fee1a2f14917923d1bb71, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:43,884 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890823883"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890823883"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890823883"}]},"ts":"1681890823883"} 2023-04-19 07:53:43,887 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:53:44,047 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:44,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3c54cc0aa26fee1a2f14917923d1bb71, NAME => 'TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.', STARTKEY => '', ENDKEY => ''} 2023-04-19 07:53:44,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:44,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:53:44,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:44,048 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:44,051 INFO [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:44,054 DEBUG [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A 2023-04-19 07:53:44,054 DEBUG [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A 2023-04-19 07:53:44,056 INFO [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:53:44,056 INFO [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3c54cc0aa26fee1a2f14917923d1bb71 columnFamilyName A 2023-04-19 07:53:44,057 INFO [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] regionserver.HStore(310): Store=3c54cc0aa26fee1a2f14917923d1bb71/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:53:44,057 INFO [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:44,060 DEBUG [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B 2023-04-19 07:53:44,060 DEBUG [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B 2023-04-19 07:53:44,060 INFO [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:53:44,061 INFO [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3c54cc0aa26fee1a2f14917923d1bb71 columnFamilyName B 2023-04-19 07:53:44,062 INFO [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] regionserver.HStore(310): Store=3c54cc0aa26fee1a2f14917923d1bb71/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:53:44,062 INFO [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:44,064 DEBUG [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C 2023-04-19 07:53:44,064 DEBUG [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C 2023-04-19 07:53:44,065 INFO [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:53:44,065 INFO [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3c54cc0aa26fee1a2f14917923d1bb71 columnFamilyName C 2023-04-19 07:53:44,066 INFO [StoreOpener-3c54cc0aa26fee1a2f14917923d1bb71-1] regionserver.HStore(310): Store=3c54cc0aa26fee1a2f14917923d1bb71/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:53:44,066 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:44,069 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:44,069 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:44,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 07:53:44,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:44,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 07:53:44,082 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3c54cc0aa26fee1a2f14917923d1bb71; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59412936, jitterRate=-0.1146782636642456}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 07:53:44,082 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:44,083 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., pid=11, masterSystemTime=1681890824040 2023-04-19 07:53:44,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:44,087 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:44,088 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=3c54cc0aa26fee1a2f14917923d1bb71, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,088 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890824088"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681890824088"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681890824088"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681890824088"}]},"ts":"1681890824088"} 2023-04-19 07:53:44,095 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-19 07:53:44,096 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 in 205 msec 2023-04-19 07:53:44,099 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-19 07:53:44,100 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3c54cc0aa26fee1a2f14917923d1bb71, ASSIGN in 369 msec 2023-04-19 07:53:44,101 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-19 07:53:44,101 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890824101"}]},"ts":"1681890824101"} 2023-04-19 07:53:44,104 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-19 07:53:44,107 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-19 07:53:44,110 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 866 msec 2023-04-19 07:53:44,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-19 07:53:44,380 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-19 07:53:44,385 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x7d9fc901 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:53:44,391 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44ef43e3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:53:44,401 DEBUG [Listener at localhost/42517] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:53:44,407 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57222, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:53:44,411 DEBUG [Listener at localhost/42517] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-19 07:53:44,414 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43782, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-19 07:53:44,423 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x79eb61dd to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:53:44,428 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61e72c30, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:53:44,428 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x748f869b to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:53:44,432 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e16d41b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:53:44,433 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x5e91d243 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:53:44,438 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fea27db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:53:44,439 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x30eb7fb2 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:53:44,443 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e728705, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:53:44,444 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x46192d17 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:53:44,448 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63859c62, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:53:44,449 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x60b8c4c7 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:53:44,453 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49a45b6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:53:44,453 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x49d71120 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:53:44,480 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fef7a21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:53:44,481 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x2879e2e1 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:53:44,486 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cd3abbd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:53:44,487 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x0d24bbb7 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:53:44,491 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aad2e18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:53:44,498 DEBUG [hconnection-0x2b3986b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:53:44,498 DEBUG [hconnection-0x12fded7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:53:44,499 DEBUG [hconnection-0x5b5f7a77-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:53:44,501 DEBUG [hconnection-0x74b74e95-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:53:44,501 DEBUG [hconnection-0x527b0319-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:53:44,502 DEBUG [hconnection-0x5f9b9bf2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:53:44,502 DEBUG [hconnection-0x37e8943a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:53:44,503 DEBUG [hconnection-0x544eb923-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:53:44,504 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57236, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:53:44,505 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57240, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:53:44,507 DEBUG [hconnection-0x2a81f259-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:53:44,508 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57268, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:53:44,512 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57254, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:53:44,514 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57262, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:53:44,522 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:53:44,524 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57294, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:53:44,524 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57282, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:53:44,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:44,536 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:53:44,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:44,559 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:44,559 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:44,560 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:44,560 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:44,561 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,561 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:44,562 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:44,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:44,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:44,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:44,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:44,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:44,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:44,568 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,568 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:44,587 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:44,587 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:44,597 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:44,599 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:44,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:44,599 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:44,602 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:44,604 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:44,604 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:44,605 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 07:53:44,606 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:44,612 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:44,612 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:44,612 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,612 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:44,612 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:44,652 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,653 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,653 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:44,654 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:44,667 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13e3d459[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:44,667 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,669 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:44,669 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:44,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:44,670 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:44,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:44,670 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,670 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:44,670 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:44,670 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:44,670 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:44,670 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:44,674 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:44,674 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,675 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,675 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:44,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:44,692 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:44,692 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:44,692 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:44,692 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:44,693 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:44,696 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,696 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:44,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,696 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:44,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:44,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,699 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:44,703 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,740 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:44,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:44,740 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:44,740 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:44,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:44,740 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:44,742 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:44,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:44,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,747 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,747 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:44,748 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:44,748 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:44,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:44,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890884796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681890824745, max timestamp=1681890824799Num uniques 859; 2023-04-19 07:53:44,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-19 07:53:44,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681890824745, max timestamp=1681890824816Num uniques 859; 2023-04-19 07:53:44,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-19 07:53:44,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681890824745, max timestamp=1681890824799Num uniques 859; 2023-04-19 07:53:44,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-19 07:53:44,848 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:44,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:44,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:44,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:44,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:44,874 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:44,874 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:44,878 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:44,878 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:44,880 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,880 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:44,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/e4f359cd9a2e4ccba791ba7c9d2be373 2023-04-19 07:53:44,882 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:44,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:44,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:44,883 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,883 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,884 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:44,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:44,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:44,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:44,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:44,888 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:44,888 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:44,892 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:44,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:44,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890884883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,895 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:44,898 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:44,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:44,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890884887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,895 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:44,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:44,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:44,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890884896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890884896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,906 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:44,907 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:44,907 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:44,908 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:44,908 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:44,908 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,908 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:44,908 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:44,911 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:44,912 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:44,913 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,913 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:44,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:44,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:44,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:44,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:44,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22aec4f7[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:44,916 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:44,916 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,918 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:44,919 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:44,921 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,921 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:44,925 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:44,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:44,925 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:44,925 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:44,925 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:44,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:44,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:44,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:44,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:44,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:44,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:44,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:44,931 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:44,931 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:44,931 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:44,931 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:44,931 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:44,939 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,939 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:44,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,939 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:44,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:44,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:44,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,941 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:44,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:44,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,943 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:44,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:44,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:44,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:44,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:44,958 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,958 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:44,958 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:44,958 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:44,958 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,958 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:44,958 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:44,958 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:44,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:44,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:44,964 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:44,964 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:44,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:44,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:44,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:44,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:44,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890884964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:45,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890885000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:45,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890885003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:45,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890885011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:45,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890885011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,064 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:45,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:45,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:45,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:45,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:45,097 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:45,098 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:45,099 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:45,099 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:45,100 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,100 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,104 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,104 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:45,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:45,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:45,109 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,109 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:45,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:45,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:45,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:45,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:45,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:45,129 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:45,129 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:45,130 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:45,130 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:45,130 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:45,130 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:45,135 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:45,135 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:45,135 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,139 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:45,139 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,142 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:45,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:45,143 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,143 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:45,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:45,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@650953da[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:45,146 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:45,146 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,149 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,149 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:45,149 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:45,149 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,149 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:45,149 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:45,153 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:45,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:45,163 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:45,163 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:45,163 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:45,163 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:45,163 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:45,166 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,166 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:45,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,166 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:45,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:45,169 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,172 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:45,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890885172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,183 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:45,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:45,183 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:45,183 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,183 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,183 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,183 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,183 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,183 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,184 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:45,184 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:45,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:45,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:45,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:45,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:45,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:45,192 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:45,192 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:45,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:45,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890885208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:45,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890885214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:45,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890885218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:45,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890885222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,292 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:45,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:45,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:45,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:45,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:45,316 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:45,316 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:45,321 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:45,321 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:45,322 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,322 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,323 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:45,323 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:45,324 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,324 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:45,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:45,331 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:45,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:45,331 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:45,331 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:45,332 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:45,333 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:45,333 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:45,333 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:45,333 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:45,333 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:45,333 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:45,337 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:45,337 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:45,337 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,337 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:45,338 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:45,342 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:45,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:45,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:45,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:45,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:45,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:45,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35006992[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:45,345 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:45,345 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,351 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,351 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,351 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:45,352 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,352 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,352 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:45,352 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:45,352 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:45,352 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:45,361 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,361 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:45,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:45,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:45,362 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:45,362 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:45,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:45,367 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:45,367 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:45,367 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:45,367 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:45,367 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:45,370 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,370 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:45,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:45,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,372 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:45,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:45,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:45,373 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:45,377 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,385 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,385 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,385 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,385 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,385 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,385 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,385 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,385 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:45,385 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:45,385 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:45,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:45,388 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:45,388 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:45,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:45,385 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:45,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:45,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:45,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890885476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,489 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:45,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:45,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:45,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:45,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:45,499 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:45,499 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:45,499 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:45,500 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:45,500 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,500 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,504 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/4113b58feb9347faa6002ab984626446 2023-04-19 07:53:45,505 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,505 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:45,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:45,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:45,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:45,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:45,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:45,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:45,508 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:45,508 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:45,509 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:45,509 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:45,509 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:45,509 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:45,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:45,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:45,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:45,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:45,512 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,514 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,514 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:45,515 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:45,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:45,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890885516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:45,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:45,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38cc3277[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:45,520 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:45,521 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,522 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,523 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:45,523 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:45,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890885520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:45,525 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:45,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:45,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890885526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,528 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:45,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:45,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:45,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:45,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:45,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:45,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890885528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:45,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:45,542 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:45,542 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:45,542 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:45,542 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:45,542 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:45,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:45,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:45,547 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,550 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,556 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:45,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:45,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:45,556 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:45,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:45,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:45,557 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:45,558 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,559 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,560 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:45,561 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,562 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,562 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,562 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:45,562 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,562 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:45,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:45,571 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/53c3a03667a247c2a346b7e75bd6c413 2023-04-19 07:53:45,583 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/e4f359cd9a2e4ccba791ba7c9d2be373 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e4f359cd9a2e4ccba791ba7c9d2be373 2023-04-19 07:53:45,596 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e4f359cd9a2e4ccba791ba7c9d2be373, entries=150, sequenceid=12, filesize=11.6 K 2023-04-19 07:53:45,603 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/4113b58feb9347faa6002ab984626446 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4113b58feb9347faa6002ab984626446 2023-04-19 07:53:45,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4113b58feb9347faa6002ab984626446, entries=150, sequenceid=12, filesize=11.6 K 2023-04-19 07:53:45,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/53c3a03667a247c2a346b7e75bd6c413 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/53c3a03667a247c2a346b7e75bd6c413 2023-04-19 07:53:45,648 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/53c3a03667a247c2a346b7e75bd6c413, entries=150, sequenceid=12, filesize=11.6 K 2023-04-19 07:53:45,662 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.28 KB/129315 for 3c54cc0aa26fee1a2f14917923d1bb71 in 1057ms, sequenceid=12, compaction requested=false 2023-04-19 07:53:45,662 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:45,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:45,667 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-19 07:53:45,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:45,674 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:45,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:45,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:45,676 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:45,676 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:45,684 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:45,685 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:45,688 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,689 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:45,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:45,701 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:45,701 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:45,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:45,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:45,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:45,715 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:45,715 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:45,716 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:45,731 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:45,731 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:45,731 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:45,732 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:45,732 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=126.28 KB heapSize=362.32 KB 2023-04-19 07:53:45,732 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:45,733 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:45,733 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:45,733 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:45,733 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:45,733 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:45,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:45,988 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:46,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890886055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890886055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890886057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890886058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890886059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890886161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890886162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890886179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890886180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890886182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,236 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/0626f31d96fd4c33a2e9fc64b1b65201 2023-04-19 07:53:46,332 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/46c5a88051be4033874f4210447310e7 2023-04-19 07:53:46,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890886392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890886392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890886392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890886394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890886410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890886699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890886701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890886703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890886712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:46,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890886716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,844 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/7b65e089aa37451a8fbeb6b6c5e46782 2023-04-19 07:53:46,866 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/0626f31d96fd4c33a2e9fc64b1b65201 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/0626f31d96fd4c33a2e9fc64b1b65201 2023-04-19 07:53:46,883 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/0626f31d96fd4c33a2e9fc64b1b65201, entries=150, sequenceid=48, filesize=11.6 K 2023-04-19 07:53:46,885 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/46c5a88051be4033874f4210447310e7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/46c5a88051be4033874f4210447310e7 2023-04-19 07:53:46,896 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/46c5a88051be4033874f4210447310e7, entries=150, sequenceid=48, filesize=11.6 K 2023-04-19 07:53:46,906 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/7b65e089aa37451a8fbeb6b6c5e46782 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7b65e089aa37451a8fbeb6b6c5e46782 2023-04-19 07:53:46,919 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7b65e089aa37451a8fbeb6b6c5e46782, entries=150, sequenceid=48, filesize=11.6 K 2023-04-19 07:53:46,920 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~126.28 KB/129315, heapSize ~362.27 KB/370968, currentSize=53.67 KB/54960 for 3c54cc0aa26fee1a2f14917923d1bb71 in 1188ms, sequenceid=48, compaction requested=false 2023-04-19 07:53:46,922 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:46,922 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:46,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:46,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:46,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:46,925 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:46,925 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:46,927 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:46,927 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:46,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:46,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c11d0e5[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:46,931 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:46,931 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:46,932 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:46,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:46,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:46,932 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:46,932 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:46,932 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:46,933 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,933 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:46,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:46,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:46,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:46,935 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:46,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:46,942 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:46,942 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:46,942 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:46,942 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:46,942 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:46,945 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:46,945 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:46,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:46,946 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:46,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:46,947 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:46,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:46,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:46,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:46,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,950 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:46,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:46,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:46,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:46,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:46,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:46,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:46,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:46,961 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:46,961 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:46,961 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:46,961 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:46,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:46,961 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:46,961 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:46,961 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:47,034 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-19 07:53:47,063 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:47,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:47,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:47,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:47,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:47,076 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:47,076 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:47,081 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:47,081 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:47,081 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:47,082 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:47,084 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,084 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:47,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:47,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:47,084 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,084 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:47,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:47,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:47,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:47,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:47,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:47,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:47,087 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:47,087 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:47,088 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:47,088 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:47,088 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:47,089 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:47,089 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:47,089 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 07:53:47,093 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:47,093 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:47,093 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:47,093 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:47,093 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:47,093 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:47,149 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-19 07:53:47,150 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-19 07:53:47,220 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:47,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:47,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890887265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890887266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890887267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890827253, max timestamp=1681890827267Num uniques 859; 2023-04-19 07:53:47,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:53:47,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890827253, max timestamp=1681890827266Num uniques 859; 2023-04-19 07:53:47,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:53:47,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890827253, max timestamp=1681890827266Num uniques 859; 2023-04-19 07:53:47,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:53:47,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890887354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890887357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890887384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890887394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890887394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890887470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890887470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890887589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890887603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890887604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,609 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/fb0daf70a1d74aeb997822cdc7c59dc2 2023-04-19 07:53:47,678 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/5534937eca1342fa96cf57273b97c327 2023-04-19 07:53:47,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890887678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890887679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,717 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/96c66aaeaa1f4985a8f4df8ecbf4a37d 2023-04-19 07:53:47,729 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/fb0daf70a1d74aeb997822cdc7c59dc2 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/fb0daf70a1d74aeb997822cdc7c59dc2 2023-04-19 07:53:47,739 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/fb0daf70a1d74aeb997822cdc7c59dc2, entries=100, sequenceid=59, filesize=9.3 K 2023-04-19 07:53:47,745 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/5534937eca1342fa96cf57273b97c327 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/5534937eca1342fa96cf57273b97c327 2023-04-19 07:53:47,759 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/5534937eca1342fa96cf57273b97c327, entries=100, sequenceid=59, filesize=9.3 K 2023-04-19 07:53:47,762 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/96c66aaeaa1f4985a8f4df8ecbf4a37d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/96c66aaeaa1f4985a8f4df8ecbf4a37d 2023-04-19 07:53:47,790 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/96c66aaeaa1f4985a8f4df8ecbf4a37d, entries=100, sequenceid=59, filesize=9.3 K 2023-04-19 07:53:47,795 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3c54cc0aa26fee1a2f14917923d1bb71 in 706ms, sequenceid=59, compaction requested=true 2023-04-19 07:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:47,796 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:47,797 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:47,797 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,797 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:47,797 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:47,799 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:47,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:47,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43a9d93e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:47,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:47,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:47,805 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:47,806 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:47,808 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:47,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:47,808 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:47,808 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:47,808 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,808 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:47,808 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:47,810 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:47,810 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:47,810 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:47,810 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:47,819 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:47,819 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:47,819 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:47,819 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:47,819 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:47,824 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:47,824 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:47,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:47,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:47,825 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:47,828 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:47,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,831 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,835 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,835 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:47,836 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:47,836 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:47,836 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:47,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:47,836 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,836 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:47,836 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:47,836 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:47,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:47,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:47,837 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:47,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:47,841 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:47,841 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:47,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:47,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-19 07:53:47,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:47,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:47,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:47,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:47,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:47,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:47,941 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:47,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:47,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:47,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890887947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890887957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890887957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:47,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:47,977 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:47,977 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:47,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890887986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:47,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890887987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:47,994 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:47,994 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:47,995 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:47,995 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,002 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:48,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:48,003 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,006 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,006 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:48,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:48,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:48,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:48,019 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:48,019 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:48,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:48,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:48,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:48,021 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:48,021 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:48,021 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:48,021 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:48,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:48,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:48,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:48,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:48,025 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,025 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,027 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,027 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:48,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75d37420[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:48,036 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:48,036 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,038 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:48,038 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,038 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:48,038 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:48,038 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:48,040 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:48,040 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:48,040 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:48,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:48,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:48,059 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:48,059 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:48,059 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:48,059 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:48,059 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:48,062 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,063 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:48,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,064 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:48,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:48,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:48,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890888058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890888061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,071 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:48,078 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890888061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,100 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,100 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,100 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,100 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,100 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,100 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:48,100 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:48,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:48,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:48,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:48,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:48,100 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,101 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,100 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:48,101 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:48,104 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:48,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:48,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:48,151 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-19 07:53:48,152 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-19 07:53:48,204 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:48,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:48,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:48,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:48,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:48,220 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:48,220 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:48,221 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:48,221 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:48,224 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,224 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,226 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:48,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:48,226 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,227 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,227 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:48,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:48,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:48,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:48,229 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:48,229 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:48,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:48,238 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:48,240 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:48,240 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:48,240 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:48,240 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:48,240 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:48,240 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:48,240 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,240 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:48,244 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,245 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,246 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:48,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c55bf67[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:48,250 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:48,250 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,255 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,255 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:48,255 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:48,256 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:48,256 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,256 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:48,257 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,258 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:48,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:48,264 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:48,264 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:48,264 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:48,264 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:48,264 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:48,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:48,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,274 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:48,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890888274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,279 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:48,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890888277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:48,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:48,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:48,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:48,287 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:48,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:48,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:48,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:48,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,291 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:48,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:48,291 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:48,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:48,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890888297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,392 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:48,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:48,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:48,409 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/5af29cb5a72349c888437dc69e9bd592 2023-04-19 07:53:48,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:48,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:48,413 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:48,413 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:48,414 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:48,415 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:48,415 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,415 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,418 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:48,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:48,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:48,419 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,420 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,420 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:48,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:48,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:48,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:48,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:48,424 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:48,424 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:48,427 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:48,428 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:48,428 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:48,428 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:48,428 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:48,429 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,429 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:48,429 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:48,431 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,431 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,432 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,432 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:48,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ea1d4f0[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:48,436 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:48,436 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,443 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,443 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:48,443 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:48,443 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:48,446 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,446 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:48,446 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:48,446 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,446 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:48,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,447 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:48,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:48,454 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:48,454 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:48,454 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:48,454 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:48,454 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:48,458 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,458 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:48,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:48,459 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,462 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,465 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:48,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:48,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:48,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:48,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:48,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:48,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,470 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:48,470 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,470 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:48,470 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:48,470 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,470 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:48,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890888498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,508 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/514bbcf6beed4267b3c876e824b956e0 2023-04-19 07:53:48,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:48,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890888500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,570 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:48,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:48,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:48,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:48,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:48,580 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:48,580 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:48,585 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:48,585 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:48,586 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,586 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,588 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:48,588 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:48,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:48,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890888586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:48,592 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,592 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:48,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:48,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:48,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890888586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:48,598 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:48,599 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:48,599 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:48,600 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:48,600 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:48,600 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:48,600 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:48,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:48,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:48,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:48,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:48,602 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:48,606 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:48,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890888605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:48,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@434c9433[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:48,611 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:48,611 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,614 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:48,614 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,614 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:48,614 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:48,616 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,616 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:48,616 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:48,616 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:48,628 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:48,628 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:48,628 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:48,628 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:48,629 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:48,633 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,633 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:48,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,634 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,636 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,646 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,649 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:48,650 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:48,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:48,650 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,650 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,650 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,650 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,650 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,650 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,650 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:48,657 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:48,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:48,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:48,659 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:48,659 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:48,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:48,759 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:48,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:48,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:48,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:48,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:48,806 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:48,807 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:48,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:48,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:48,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,815 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:48,815 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:48,816 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,816 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:48,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:48,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:48,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:48,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:48,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:48,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:48,836 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:48,836 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:48,836 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:48,836 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,842 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,843 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,843 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:48,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d508a0e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:48,845 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:48,845 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,847 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,847 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:48,847 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:48,847 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,847 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:48,847 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:48,849 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:48,849 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,849 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,849 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:48,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:48,852 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:48,853 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:48,853 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:48,853 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:48,853 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:48,855 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,855 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,856 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,856 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,858 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,862 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:48,862 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:48,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:48,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:48,862 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:48,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:48,862 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,865 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,865 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,865 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,865 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,865 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,865 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:48,865 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:48,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:48,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:48,939 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/e1f665d0703e4a39b73b872c56b0a9ba 2023-04-19 07:53:48,951 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/5af29cb5a72349c888437dc69e9bd592 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5af29cb5a72349c888437dc69e9bd592 2023-04-19 07:53:48,963 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5af29cb5a72349c888437dc69e9bd592, entries=200, sequenceid=96, filesize=13.9 K 2023-04-19 07:53:48,965 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/514bbcf6beed4267b3c876e824b956e0 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/514bbcf6beed4267b3c876e824b956e0 2023-04-19 07:53:48,965 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:48,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:48,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:48,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:48,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:48,975 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:48,975 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:48,976 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:48,976 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:48,977 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,977 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:48,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,980 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:48,980 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,981 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:48,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:48,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:48,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:48,984 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:48,984 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:48,985 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:48,987 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:48,987 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:48,988 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:48,988 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:48,988 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:48,988 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:48,990 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/514bbcf6beed4267b3c876e824b956e0, entries=150, sequenceid=96, filesize=11.6 K 2023-04-19 07:53:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:48,990 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:48,991 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,991 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:48,993 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/e1f665d0703e4a39b73b872c56b0a9ba as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/e1f665d0703e4a39b73b872c56b0a9ba 2023-04-19 07:53:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:48,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:48,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c7fb644[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:48,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:48,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,999 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:48,999 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:48,999 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:48,999 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:49,000 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,000 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:49,002 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,002 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,002 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:49,002 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:49,009 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/e1f665d0703e4a39b73b872c56b0a9ba, entries=150, sequenceid=96, filesize=11.6 K 2023-04-19 07:53:49,010 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for 3c54cc0aa26fee1a2f14917923d1bb71 in 1104ms, sequenceid=96, compaction requested=true 2023-04-19 07:53:49,011 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,018 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,018 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:53:49,018 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:53:49,018 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:53:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,019 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:53:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:49,022 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:49,022 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:49,022 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:49,022 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:49,022 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:49,027 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:53:49,029 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,029 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,030 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/B is initiating minor compaction (all files) 2023-04-19 07:53:49,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,030 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,030 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/B in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:49,030 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,030 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4113b58feb9347faa6002ab984626446, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/46c5a88051be4033874f4210447310e7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/5534937eca1342fa96cf57273b97c327, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/514bbcf6beed4267b3c876e824b956e0] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=44.1 K 2023-04-19 07:53:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:49,031 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47540 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:53:49,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:49,032 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/A is initiating minor compaction (all files) 2023-04-19 07:53:49,032 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/A in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:49,032 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e4f359cd9a2e4ccba791ba7c9d2be373, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/0626f31d96fd4c33a2e9fc64b1b65201, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/fb0daf70a1d74aeb997822cdc7c59dc2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5af29cb5a72349c888437dc69e9bd592] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=46.4 K 2023-04-19 07:53:49,034 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 4113b58feb9347faa6002ab984626446, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681890824537 2023-04-19 07:53:49,034 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting e4f359cd9a2e4ccba791ba7c9d2be373, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681890824537 2023-04-19 07:53:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,041 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 46c5a88051be4033874f4210447310e7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681890824880 2023-04-19 07:53:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,042 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 5534937eca1342fa96cf57273b97c327, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681890826054 2023-04-19 07:53:49,043 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 0626f31d96fd4c33a2e9fc64b1b65201, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681890824880 2023-04-19 07:53:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,043 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,044 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting fb0daf70a1d74aeb997822cdc7c59dc2, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681890826054 2023-04-19 07:53:49,045 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 514bbcf6beed4267b3c876e824b956e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681890827326 2023-04-19 07:53:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,045 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 5af29cb5a72349c888437dc69e9bd592, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681890827326 2023-04-19 07:53:49,047 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:49,047 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:49,047 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:49,047 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,048 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:49,048 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,048 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,048 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,048 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,048 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:49,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:49,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:49,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:49,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:49,051 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:49,051 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:49,099 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#B#compaction#12 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:49,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:49,105 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 07:53:49,112 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:49,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:49,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:49,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:49,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:49,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:49,118 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#A#compaction#13 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:49,154 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:49,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:49,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:49,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:49,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:49,181 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:49,181 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:49,186 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:49,187 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:49,190 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,190 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,190 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/1cdd7fce798a4f73a47bf7d88060f5dd 2023-04-19 07:53:49,198 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:49,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:49,198 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,199 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,199 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:49,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:49,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:49,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:49,200 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:49,201 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:49,201 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:49,202 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:49,202 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:49,202 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:49,202 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:49,202 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:49,205 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,205 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,205 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,205 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:49,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c39e16f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:49,209 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:49,209 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,211 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,211 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:49,211 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:49,211 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:49,212 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,212 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:49,214 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,214 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,214 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:49,214 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:49,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:49,219 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:49,219 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:49,219 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:49,219 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:49,219 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:49,221 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,221 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:49,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,222 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,227 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,231 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,236 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,236 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,236 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,236 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,236 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,236 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:49,236 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:49,236 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:49,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:49,239 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:49,239 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:49,240 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:49,240 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:49,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:49,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681890829206, max timestamp=1681890829235Num uniques 859; 2023-04-19 07:53:49,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:53:49,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890829206, max timestamp=1681890829235Num uniques 859; 2023-04-19 07:53:49,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681890829218, max timestamp=1681890829235Num uniques 859; 2023-04-19 07:53:49,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-19 07:53:49,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:53:49,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:49,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890889320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:49,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890889321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:49,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890889322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,340 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:49,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:49,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:49,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:49,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:49,347 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:49,347 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:49,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:49,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:49,349 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,349 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,350 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:49,350 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:49,351 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,351 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:49,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:49,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:49,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:49,352 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:49,352 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:49,353 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:49,354 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:49,354 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:49,354 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:49,354 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:49,354 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:49,354 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:49,354 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:49,354 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:49,354 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,354 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:49,354 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:49,357 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,357 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,357 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,357 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:49,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:49,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b730dd7[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:49,359 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:49,359 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,363 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,363 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:49,363 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:49,363 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,363 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:49,363 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:49,365 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:49,365 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,365 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:49,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,365 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:49,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:49,370 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:49,370 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:49,370 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:49,370 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:49,370 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:49,372 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,372 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:49,372 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,374 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,378 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,384 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:49,384 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:49,384 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:49,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:49,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:49,384 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,386 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,386 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,386 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,386 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,386 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,384 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:49,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:49,386 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:49,387 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:49,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890889429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:49,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890889430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:49,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890889430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,487 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:49,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:49,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:49,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:49,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:49,497 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:49,497 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:49,498 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:49,498 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:49,499 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,499 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,504 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:49,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:49,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:49,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:49,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:49,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,507 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:49,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:49,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:49,508 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:49,508 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:49,513 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:49,514 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:49,514 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:49,514 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:49,515 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:49,515 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:49,517 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:49,518 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:49,518 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:49,518 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,518 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:49,518 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:49,521 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,521 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,522 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,522 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:49,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890889519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:49,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890889521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:49,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@625aee4d[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:49,526 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:49,526 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,528 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:49,529 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,529 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:49,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:49,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:49,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:49,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:49,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:49,548 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:49,548 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:49,548 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:49,548 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:49,549 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:49,551 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,551 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:49,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:49,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:49,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,559 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,565 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,566 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,569 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:49,569 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:49,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:49,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:49,570 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,570 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,570 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,570 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,570 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,570 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,570 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:49,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:49,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:49,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:49,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:49,580 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:49,580 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:49,604 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/be3abaab292642959b61df83a645afb4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/be3abaab292642959b61df83a645afb4 2023-04-19 07:53:49,604 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/b2220ff6c478474f89a4641a22f44fb7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b2220ff6c478474f89a4641a22f44fb7 2023-04-19 07:53:49,635 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/B of 3c54cc0aa26fee1a2f14917923d1bb71 into be3abaab292642959b61df83a645afb4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:53:49,635 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:49,635 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/B, priority=12, startTime=1681890829018; duration=0sec 2023-04-19 07:53:49,636 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:53:49,636 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:53:49,637 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/A of 3c54cc0aa26fee1a2f14917923d1bb71 into b2220ff6c478474f89a4641a22f44fb7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:53:49,637 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:49,638 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/A, priority=12, startTime=1681890829017; duration=0sec 2023-04-19 07:53:49,638 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:49,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:49,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890889637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,643 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:53:49,643 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/C is initiating minor compaction (all files) 2023-04-19 07:53:49,643 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/C in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:49,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:49,644 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/53c3a03667a247c2a346b7e75bd6c413, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7b65e089aa37451a8fbeb6b6c5e46782, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/96c66aaeaa1f4985a8f4df8ecbf4a37d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/e1f665d0703e4a39b73b872c56b0a9ba] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=44.1 K 2023-04-19 07:53:49,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890889641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:49,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890889641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,645 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 53c3a03667a247c2a346b7e75bd6c413, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681890824537 2023-04-19 07:53:49,649 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/3789208365b64fe2b58d2b40f1961e8e 2023-04-19 07:53:49,651 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7b65e089aa37451a8fbeb6b6c5e46782, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681890824880 2023-04-19 07:53:49,656 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 96c66aaeaa1f4985a8f4df8ecbf4a37d, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681890826054 2023-04-19 07:53:49,656 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting e1f665d0703e4a39b73b872c56b0a9ba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681890827326 2023-04-19 07:53:49,680 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:49,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:49,683 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#C#compaction#17 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:49,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:49,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:49,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:49,687 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:49,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:49,692 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:49,693 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:49,693 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,693 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,695 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,695 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:49,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:49,696 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,696 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:49,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:49,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:49,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:49,706 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:49,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:49,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:49,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:49,710 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:49,710 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:49,711 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:49,711 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:49,711 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:49,716 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,716 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,717 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,717 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:49,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:49,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3346a9d4[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:49,721 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:49,721 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,722 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,723 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:49,723 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,723 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:49,723 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:49,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:49,727 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,727 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,727 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:49,727 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,731 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/0d3df69d957b4e5dbc9862a959495fc8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0d3df69d957b4e5dbc9862a959495fc8 2023-04-19 07:53:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:49,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:49,735 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:49,735 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:49,735 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:49,735 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:49,735 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:49,738 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,738 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:49,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,738 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:49,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:49,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,739 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,742 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/C of 3c54cc0aa26fee1a2f14917923d1bb71 into 0d3df69d957b4e5dbc9862a959495fc8(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:53:49,743 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:49,743 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/C, priority=12, startTime=1681890829018; duration=0sec 2023-04-19 07:53:49,743 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,744 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,747 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:49,747 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:49,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:49,747 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,748 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:49,748 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:49,753 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,753 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,753 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,753 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,753 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:49,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:49,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:49,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:49,754 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:49,754 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:49,854 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:49,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:49,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:49,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:49,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:49,865 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:49,866 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:49,867 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:49,867 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:49,867 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,867 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,868 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:49,869 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,871 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,872 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:49,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:49,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:49,882 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:49,882 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:49,882 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:49,882 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:49,883 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:49,883 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:49,883 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:49,883 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:49,883 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:49,883 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,883 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:49,883 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:49,885 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,885 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,886 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,886 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:49,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:49,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10d7044a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:49,888 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:49,888 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,889 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:49,890 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:49,890 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,890 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:49,892 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:49,892 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,892 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,892 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:49,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:49,898 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:49,898 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:49,898 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:49,898 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:49,898 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:49,905 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,905 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:49,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,905 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:49,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,917 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,919 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,922 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:49,922 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,922 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:49,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:49,922 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:49,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:49,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:49,924 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:49,924 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:49,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:49,922 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,924 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:49,924 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,924 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,924 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:49,924 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:49,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:49,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:49,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890889945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:49,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890889945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:49,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:49,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890889947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,024 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:50,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:50,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:50,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:50,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:50,029 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:50,029 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:50,030 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:50,030 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:50,031 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,031 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:50,034 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:50,034 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:50,034 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,034 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:50,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:50,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:50,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:50,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:50,039 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:50,039 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:50,040 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:50,040 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:50,040 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:50,040 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:50,042 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,042 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:50,042 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,042 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:50,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:50,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4399be32[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:50,046 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:50,046 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,051 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,051 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:50,052 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:50,052 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:50,056 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,056 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:50,057 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:50,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:50,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:50,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:50,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:50,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:50,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:50,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:50,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:50,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:50,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:50,063 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:50,063 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:50,063 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:50,063 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:50,063 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:50,066 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,066 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:50,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:50,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,066 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:50,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:50,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:50,067 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:50,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:50,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:50,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,070 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,073 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:50,073 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:50,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:50,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:50,073 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:50,073 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:50,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,073 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:50,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:50,073 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:50,074 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:50,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:50,075 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:50,115 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/28c0d9922233432d8b48c6fcbd537a4b 2023-04-19 07:53:50,127 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/1cdd7fce798a4f73a47bf7d88060f5dd as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/1cdd7fce798a4f73a47bf7d88060f5dd 2023-04-19 07:53:50,137 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/1cdd7fce798a4f73a47bf7d88060f5dd, entries=200, sequenceid=108, filesize=13.9 K 2023-04-19 07:53:50,139 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/3789208365b64fe2b58d2b40f1961e8e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/3789208365b64fe2b58d2b40f1961e8e 2023-04-19 07:53:50,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/3789208365b64fe2b58d2b40f1961e8e, entries=150, sequenceid=108, filesize=11.6 K 2023-04-19 07:53:50,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/28c0d9922233432d8b48c6fcbd537a4b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/28c0d9922233432d8b48c6fcbd537a4b 2023-04-19 07:53:50,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/28c0d9922233432d8b48c6fcbd537a4b, entries=150, sequenceid=108, filesize=11.6 K 2023-04-19 07:53:50,170 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-19 07:53:50,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=122.60 KB/125545 for 3c54cc0aa26fee1a2f14917923d1bb71 in 1068ms, sequenceid=108, compaction requested=false 2023-04-19 07:53:50,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:50,175 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:50,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:50,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:50,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:50,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:50,184 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:50,184 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:50,185 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:50,185 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:50,186 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,186 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:50,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:50,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:50,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:50,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:50,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:50,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:50,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:50,190 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,190 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:50,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:50,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:50,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:50,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:50,199 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:50,200 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:50,200 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:50,202 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=122.60 KB heapSize=344.62 KB 2023-04-19 07:53:50,202 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:50,202 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:50,202 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:50,203 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:50,203 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:50,203 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:50,248 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/526fae089dc648a9aa15c579d2de0881 2023-04-19 07:53:50,315 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/5bae100107ec4813bcf91b97100e874b 2023-04-19 07:53:50,453 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:50,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:50,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:50,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890890469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:50,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890890470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:50,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890890475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:50,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:50,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890890577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890890578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:50,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890890581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,771 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=37.62 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/4562290cf5434c2bbec342791ab1c778 2023-04-19 07:53:50,782 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/526fae089dc648a9aa15c579d2de0881 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/526fae089dc648a9aa15c579d2de0881 2023-04-19 07:53:50,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:50,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:50,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890890785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,794 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/526fae089dc648a9aa15c579d2de0881, entries=150, sequenceid=145, filesize=11.8 K 2023-04-19 07:53:50,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:50,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890890785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890890784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,798 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/5bae100107ec4813bcf91b97100e874b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/5bae100107ec4813bcf91b97100e874b 2023-04-19 07:53:50,816 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/5bae100107ec4813bcf91b97100e874b, entries=150, sequenceid=145, filesize=11.8 K 2023-04-19 07:53:50,817 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/4562290cf5434c2bbec342791ab1c778 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/4562290cf5434c2bbec342791ab1c778 2023-04-19 07:53:50,829 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/4562290cf5434c2bbec342791ab1c778, entries=150, sequenceid=145, filesize=11.8 K 2023-04-19 07:53:50,830 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~122.60 KB/125545, heapSize ~344.57 KB/352840, currentSize=60.38 KB/61830 for 3c54cc0aa26fee1a2f14917923d1bb71 in 630ms, sequenceid=145, compaction requested=true 2023-04-19 07:53:50,830 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:50,830 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:50,831 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:50,831 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:50,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:50,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:50,835 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,835 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:50,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7684088a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:50,839 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:50,839 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,840 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:50,840 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:50,840 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:50,841 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,841 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:50,842 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:50,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:50,847 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:50,847 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:50,847 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:50,847 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:50,847 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:50,848 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,848 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:50,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,848 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:50,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,851 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,853 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,857 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:50,857 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:50,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:50,858 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:50,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:50,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:50,858 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:50,858 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:50,858 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,858 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:50,858 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:50,858 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:50,858 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,861 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:50,861 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,958 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:50,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:50,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:50,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:50,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:50,964 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:50,964 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:50,964 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:50,964 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:50,965 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,965 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:50,967 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:50,967 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:50,968 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:50,968 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:50,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:50,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:50,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:50,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:50,970 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:50,970 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:50,970 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:50,972 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:50,972 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:50,972 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:50,972 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:50,972 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 07:53:50,973 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:50,973 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:50,973 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:50,973 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:50,973 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:50,973 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:51,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:51,101 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:51,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890831116, max timestamp=1681890831127Num uniques 859; 2023-04-19 07:53:51,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890831116, max timestamp=1681890831127Num uniques 859; 2023-04-19 07:53:51,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:53:51,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:53:51,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890831116, max timestamp=1681890831127Num uniques 859; 2023-04-19 07:53:51,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:53:51,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:51,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890891174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:51,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890891174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:51,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890891175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:51,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890891280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:51,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890891281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:51,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890891281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,396 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/6daf8d58af844023b3745815f67e872c 2023-04-19 07:53:51,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:51,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890891486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:51,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890891487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:51,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890891488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:51,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890891527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:51,532 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:53:51,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890891530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,542 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:53:51,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:51,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890891793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:51,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890891794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:51,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890891794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,834 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/8ac386c0987a4885a031631f4136bbab 2023-04-19 07:53:51,857 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/ff6a0f92720c4621827579aa3a12e596 2023-04-19 07:53:51,864 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/6daf8d58af844023b3745815f67e872c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/6daf8d58af844023b3745815f67e872c 2023-04-19 07:53:51,871 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/6daf8d58af844023b3745815f67e872c, entries=150, sequenceid=157, filesize=11.8 K 2023-04-19 07:53:51,872 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/8ac386c0987a4885a031631f4136bbab as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/8ac386c0987a4885a031631f4136bbab 2023-04-19 07:53:51,898 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/8ac386c0987a4885a031631f4136bbab, entries=150, sequenceid=157, filesize=11.8 K 2023-04-19 07:53:51,900 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/ff6a0f92720c4621827579aa3a12e596 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/ff6a0f92720c4621827579aa3a12e596 2023-04-19 07:53:51,908 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/ff6a0f92720c4621827579aa3a12e596, entries=150, sequenceid=157, filesize=11.8 K 2023-04-19 07:53:51,910 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=134.18 KB/137400 for 3c54cc0aa26fee1a2f14917923d1bb71 in 938ms, sequenceid=157, compaction requested=true 2023-04-19 07:53:51,910 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:51,910 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:51,910 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:51,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:51,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:51,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:51,913 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:51,913 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:51,914 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:51,914 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:51,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:51,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@708a1787[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:51,915 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:51,915 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:51,916 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:51,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:51,916 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:51,916 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:51,916 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:51,916 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,917 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:51,919 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,919 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:51,919 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:51,919 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:51,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:51,925 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:51,925 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:51,925 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:51,925 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:51,925 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:51,932 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:51,932 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:51,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:51,932 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:51,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:51,934 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:51,936 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:51,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:51,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:51,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:51,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:51,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:51,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:51,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:51,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:51,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:51,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:51,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:51,944 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:51,944 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:51,945 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:51,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:52,045 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:52,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:52,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:52,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:52,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:52,054 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:52,054 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:52,054 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:52,054 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:52,054 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:52,054 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:52,057 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:52,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:52,058 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:52,058 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:52,058 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:52,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:52,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:52,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:52,060 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:52,060 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:52,060 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:52,060 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:52,061 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:52,061 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:52,061 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:52,061 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-19 07:53:52,061 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:52,062 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:52,062 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:52,062 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:52,062 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:52,062 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:52,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:52,298 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:52,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:52,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890892316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:52,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:52,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890892318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:52,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:52,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890892319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:52,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:52,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890892421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:52,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:52,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890892423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:52,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:52,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890892425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:52,510 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/7709fcfcad6c4be997206ed7dcda77ff 2023-04-19 07:53:52,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:52,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890892623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:52,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:52,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890892629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:52,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:52,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890892633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:52,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:52,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890892929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:52,932 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/4a4defb796164216b4aa91a18437962a 2023-04-19 07:53:52,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:52,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890892932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:52,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:52,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890892938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:52,954 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/7669c97a72014713a233eefd8ca4b449 2023-04-19 07:53:52,962 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/7709fcfcad6c4be997206ed7dcda77ff as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/7709fcfcad6c4be997206ed7dcda77ff 2023-04-19 07:53:52,970 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/7709fcfcad6c4be997206ed7dcda77ff, entries=150, sequenceid=193, filesize=11.8 K 2023-04-19 07:53:52,973 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/4a4defb796164216b4aa91a18437962a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4a4defb796164216b4aa91a18437962a 2023-04-19 07:53:52,982 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4a4defb796164216b4aa91a18437962a, entries=150, sequenceid=193, filesize=11.8 K 2023-04-19 07:53:52,984 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/7669c97a72014713a233eefd8ca4b449 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7669c97a72014713a233eefd8ca4b449 2023-04-19 07:53:52,997 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7669c97a72014713a233eefd8ca4b449, entries=150, sequenceid=193, filesize=11.8 K 2023-04-19 07:53:52,999 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 3c54cc0aa26fee1a2f14917923d1bb71 in 938ms, sequenceid=193, compaction requested=true 2023-04-19 07:53:52,999 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:52,999 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:53,000 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:53,001 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:53,001 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,002 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:53,002 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:53,003 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,004 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,004 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:53,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5857a888[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:53,010 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:53,010 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,011 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,011 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:53,011 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:53,011 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:53,013 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,013 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:53,014 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:53,014 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:53,014 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,014 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:53,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:53,017 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:53,017 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:53,017 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:53,017 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:53,017 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:53,022 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,022 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,022 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:53,023 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,024 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:53,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:53,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:53,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:53,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:53,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:53,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:53,029 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:53,029 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:53,028 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:53,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:53,129 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:53,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:53,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:53,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:53,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:53,136 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:53,136 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:53,136 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:53,136 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:53,137 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,137 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,138 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:53,138 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:53,139 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,139 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:53,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:53,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:53,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:53,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:53,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:53,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:53,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:53,141 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:53,141 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:53,141 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:53:53,142 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:53,143 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:53,143 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:53,144 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:53,144 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:53,144 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:53,159 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/e36897b2d41548c5839fb4405a9a53c6 2023-04-19 07:53:53,181 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/c96e0b27feb54c8586994e93afca2b2c 2023-04-19 07:53:53,209 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/06525ba9416740038df3865fba66447d 2023-04-19 07:53:53,219 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/e36897b2d41548c5839fb4405a9a53c6 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e36897b2d41548c5839fb4405a9a53c6 2023-04-19 07:53:53,229 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e36897b2d41548c5839fb4405a9a53c6, entries=150, sequenceid=203, filesize=11.8 K 2023-04-19 07:53:53,236 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/c96e0b27feb54c8586994e93afca2b2c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/c96e0b27feb54c8586994e93afca2b2c 2023-04-19 07:53:53,252 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/c96e0b27feb54c8586994e93afca2b2c, entries=150, sequenceid=203, filesize=11.8 K 2023-04-19 07:53:53,255 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/06525ba9416740038df3865fba66447d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/06525ba9416740038df3865fba66447d 2023-04-19 07:53:53,276 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/06525ba9416740038df3865fba66447d, entries=150, sequenceid=203, filesize=11.8 K 2023-04-19 07:53:53,281 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 3c54cc0aa26fee1a2f14917923d1bb71 in 140ms, sequenceid=203, compaction requested=true 2023-04-19 07:53:53,281 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:53,281 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:53,281 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:53,282 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:53,282 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,282 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:53,282 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:53,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:53,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5631f59e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:53,288 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:53,288 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,290 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,290 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:53,292 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,292 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,292 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:53,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,292 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:53,292 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:53,292 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:53,294 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,294 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,294 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:53,295 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:53,298 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:53,298 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:53,298 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:53,298 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:53,298 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:53,303 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,303 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,303 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,306 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,309 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,318 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,318 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:53,318 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,318 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,318 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,318 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,318 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,318 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:53,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:53,318 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:53,319 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:53,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:53,319 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:53,319 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:53,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:53,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:53,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:53,419 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:53,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:53,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:53,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:53,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:53,424 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:53,424 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:53,424 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:53,424 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:53,425 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,425 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,427 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:53,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:53,427 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:53,428 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,428 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:53,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:53,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:53,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:53,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:53,429 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:53,429 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:53,429 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:53,429 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:53,429 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:53,429 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:53,429 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:53,429 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:53,430 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:53,430 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:53,430 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:53,430 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,430 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:53,430 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:53,431 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,431 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,432 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:53,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:53,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:53,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27778a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:53,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,435 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,435 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:53,435 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:53,435 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:53,435 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,435 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:53,437 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,437 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,437 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:53,437 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:53,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:53,441 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:53,441 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:53,441 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:53,441 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:53,441 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:53,447 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,447 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,447 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:53,448 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:53,450 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:53:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,451 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:53,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:53,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:53,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:53,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:53,453 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:53,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,460 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,460 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,460 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:53,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:53,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:53,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:53,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:53,460 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:53,460 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:53,460 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:53,460 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:53,460 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:53,461 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,461 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,461 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,461 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:53,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/629e655354664747b9ed4f04535e40a7 2023-04-19 07:53:53,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/54b37c12de5d4dc8842e1ee72c004e67 2023-04-19 07:53:53,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890833465, max timestamp=1681890833506Num uniques 859; 2023-04-19 07:53:53,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:53:53,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890833465, max timestamp=1681890833506Num uniques 859; 2023-04-19 07:53:53,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:53:53,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890833465, max timestamp=1681890833506Num uniques 859; 2023-04-19 07:53:53,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:53:53,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:53,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890893538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:53,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890893541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:53,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890893543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,561 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:53,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:53,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:53,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:53,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:53,575 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:53,575 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:53,575 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:53,575 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:53,576 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,576 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,580 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:53,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,580 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,581 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,581 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:53,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:53,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:53,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:53,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:53,596 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:53,596 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:53,596 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:53,596 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:53,596 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:53,596 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:53,596 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:53,596 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:53,596 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,596 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:53,596 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:53,599 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,599 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,599 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,599 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:53,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20b16e26[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:53,601 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:53,601 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:53,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:53,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:53,603 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,603 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:53,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:53,604 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:53,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:53,609 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:53,610 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:53,610 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:53,610 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:53,610 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:53,611 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,611 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,611 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:53,612 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,616 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,620 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,620 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,620 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:53,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:53,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:53,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:53,621 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:53,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:53,620 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:53,621 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:53,621 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:53,620 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,621 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,621 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:53,621 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:53,621 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:53,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890893644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:53,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890893647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:53,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890893651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,721 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:53,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:53,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:53,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:53,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:53,727 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:53,727 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:53,727 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:53,727 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:53,728 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,728 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,735 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,735 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:53,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:53,735 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,735 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:53,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:53,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:53,737 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:53,738 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:53,738 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:53,738 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:53,738 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:53,738 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:53,739 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,739 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,740 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,740 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:53,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45c78150[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:53,742 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:53,742 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,743 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,743 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:53,743 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:53,743 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:53,743 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,743 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:53,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:53,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:53,749 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:53,749 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:53,749 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:53,749 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:53,749 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:53,751 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,751 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,751 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,751 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,753 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,756 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:53,756 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:53,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:53,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:53,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:53,756 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,756 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,756 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,756 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,756 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,756 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,756 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:53,756 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,756 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:53,756 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:53,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:53,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:53,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:53,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57322 deadline: 1681890893853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,856 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:53,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:53,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:53,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890893854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:53,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890893857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:53,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:53,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:53,861 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:53,861 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:53,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:53,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:53,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,863 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,863 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:53,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:53,864 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,864 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:53,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:53,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:53,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:53,870 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:53,870 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:53,870 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:53,870 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:53,871 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,871 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:53,871 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:53,877 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,877 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:53,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c030173[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:53,880 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:53,880 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,881 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,881 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,881 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:53,881 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:53,881 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:53,882 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:53,883 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,883 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:53,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,883 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:53,883 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:53,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:53,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:53,886 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:53,886 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:53,886 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:53,886 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:53,886 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:53,888 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,888 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,888 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:53,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:53,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:53,889 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:53,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,890 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:53,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:53,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:53,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:53,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:53,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:53,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:53,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:53,896 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:53,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:53,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:53,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:53,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:53,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:53,897 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:53,897 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:53,920 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/b69a46f736b54f97ad457d8d4b8da88d 2023-04-19 07:53:53,931 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/629e655354664747b9ed4f04535e40a7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/629e655354664747b9ed4f04535e40a7 2023-04-19 07:53:53,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/629e655354664747b9ed4f04535e40a7, entries=200, sequenceid=215, filesize=14.1 K 2023-04-19 07:53:53,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/54b37c12de5d4dc8842e1ee72c004e67 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/54b37c12de5d4dc8842e1ee72c004e67 2023-04-19 07:53:53,955 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/54b37c12de5d4dc8842e1ee72c004e67, entries=150, sequenceid=215, filesize=11.8 K 2023-04-19 07:53:53,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/b69a46f736b54f97ad457d8d4b8da88d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b69a46f736b54f97ad457d8d4b8da88d 2023-04-19 07:53:53,969 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b69a46f736b54f97ad457d8d4b8da88d, entries=150, sequenceid=215, filesize=11.8 K 2023-04-19 07:53:53,970 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3c54cc0aa26fee1a2f14917923d1bb71 in 520ms, sequenceid=215, compaction requested=true 2023-04-19 07:53:53,970 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:53,972 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:53,972 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-19 07:53:53,972 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-19 07:53:53,972 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:53:53,973 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:53:53,978 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 88820 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-19 07:53:53,978 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84090 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-19 07:53:53,978 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/A is initiating minor compaction (all files) 2023-04-19 07:53:53,978 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/B is initiating minor compaction (all files) 2023-04-19 07:53:53,978 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/A in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:53,978 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/B in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:53,978 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b2220ff6c478474f89a4641a22f44fb7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/1cdd7fce798a4f73a47bf7d88060f5dd, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/526fae089dc648a9aa15c579d2de0881, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/6daf8d58af844023b3745815f67e872c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/7709fcfcad6c4be997206ed7dcda77ff, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e36897b2d41548c5839fb4405a9a53c6, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/629e655354664747b9ed4f04535e40a7] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=86.7 K 2023-04-19 07:53:53,978 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/be3abaab292642959b61df83a645afb4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/3789208365b64fe2b58d2b40f1961e8e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/5bae100107ec4813bcf91b97100e874b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/8ac386c0987a4885a031631f4136bbab, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4a4defb796164216b4aa91a18437962a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/c96e0b27feb54c8586994e93afca2b2c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/54b37c12de5d4dc8842e1ee72c004e67] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=82.1 K 2023-04-19 07:53:53,979 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting b2220ff6c478474f89a4641a22f44fb7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681890827326 2023-04-19 07:53:53,980 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting be3abaab292642959b61df83a645afb4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681890827326 2023-04-19 07:53:53,981 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 1cdd7fce798a4f73a47bf7d88060f5dd, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681890827938 2023-04-19 07:53:53,981 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 3789208365b64fe2b58d2b40f1961e8e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681890827938 2023-04-19 07:53:53,981 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 526fae089dc648a9aa15c579d2de0881, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681890829315 2023-04-19 07:53:53,981 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 5bae100107ec4813bcf91b97100e874b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681890829315 2023-04-19 07:53:53,982 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 6daf8d58af844023b3745815f67e872c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681890830464 2023-04-19 07:53:53,982 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 8ac386c0987a4885a031631f4136bbab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681890830464 2023-04-19 07:53:53,982 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 7709fcfcad6c4be997206ed7dcda77ff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681890831154 2023-04-19 07:53:53,982 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 4a4defb796164216b4aa91a18437962a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681890831154 2023-04-19 07:53:53,983 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting e36897b2d41548c5839fb4405a9a53c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681890832309 2023-04-19 07:53:53,983 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting c96e0b27feb54c8586994e93afca2b2c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681890832309 2023-04-19 07:53:53,983 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 629e655354664747b9ed4f04535e40a7, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681890833439 2023-04-19 07:53:53,983 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 54b37c12de5d4dc8842e1ee72c004e67, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681890833439 2023-04-19 07:53:53,997 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:53,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:54,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:54,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:54,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:54,003 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:54,003 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:54,003 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:54,003 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:54,003 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:54,003 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:54,004 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#B#compaction#34 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:54,004 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#A#compaction#33 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:54,005 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:54,005 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:54,010 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,010 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:54,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:54,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:54,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:54,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:54,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:54,048 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:54,048 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:54,048 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-19 07:53:54,049 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:54,049 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:54,049 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:54,049 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:54,049 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:54,049 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:54,064 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/7d2a9c4a00d040f9983c72d42add06ec as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/7d2a9c4a00d040f9983c72d42add06ec 2023-04-19 07:53:54,075 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/A of 3c54cc0aa26fee1a2f14917923d1bb71 into 7d2a9c4a00d040f9983c72d42add06ec(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:53:54,075 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:54,075 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/A, priority=9, startTime=1681890833972; duration=0sec 2023-04-19 07:53:54,075 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:53:54,075 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-19 07:53:54,080 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84090 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-19 07:53:54,080 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/C is initiating minor compaction (all files) 2023-04-19 07:53:54,080 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/C in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:54,080 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0d3df69d957b4e5dbc9862a959495fc8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/28c0d9922233432d8b48c6fcbd537a4b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/4562290cf5434c2bbec342791ab1c778, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/ff6a0f92720c4621827579aa3a12e596, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7669c97a72014713a233eefd8ca4b449, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/06525ba9416740038df3865fba66447d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b69a46f736b54f97ad457d8d4b8da88d] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=82.1 K 2023-04-19 07:53:54,082 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 0d3df69d957b4e5dbc9862a959495fc8, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681890827326 2023-04-19 07:53:54,082 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 28c0d9922233432d8b48c6fcbd537a4b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681890827938 2023-04-19 07:53:54,083 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 4562290cf5434c2bbec342791ab1c778, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681890829315 2023-04-19 07:53:54,083 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting ff6a0f92720c4621827579aa3a12e596, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681890830464 2023-04-19 07:53:54,084 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 7669c97a72014713a233eefd8ca4b449, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681890831154 2023-04-19 07:53:54,084 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 06525ba9416740038df3865fba66447d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681890832309 2023-04-19 07:53:54,085 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting b69a46f736b54f97ad457d8d4b8da88d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681890833439 2023-04-19 07:53:54,111 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#C#compaction#36 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:54,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:54,161 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:54,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681890894172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57322 deadline: 1681890894175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57262 deadline: 1681890894176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890894277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57322 deadline: 1681890894281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890894281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,455 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/714a6ad28fc34152a6893c292b611f87 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/714a6ad28fc34152a6893c292b611f87 2023-04-19 07:53:54,462 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/B of 3c54cc0aa26fee1a2f14917923d1bb71 into 714a6ad28fc34152a6893c292b611f87(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:53:54,462 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:54,462 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/B, priority=9, startTime=1681890833972; duration=0sec 2023-04-19 07:53:54,462 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:54,483 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/5e0ab40f3b86448b9c1cb49dc1a6d6fb 2023-04-19 07:53:54,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890894481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57322 deadline: 1681890894485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890894487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,508 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/86729f33b4aa420d8243fc7aa85bf34b 2023-04-19 07:53:54,527 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/7ae649aa61ae44759882e4512d2f51ee 2023-04-19 07:53:54,528 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/5e4750d2087c49ce9f930d341d0f54fc as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/5e4750d2087c49ce9f930d341d0f54fc 2023-04-19 07:53:54,539 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/C of 3c54cc0aa26fee1a2f14917923d1bb71 into 5e4750d2087c49ce9f930d341d0f54fc(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:53:54,539 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:54,539 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/C, priority=9, startTime=1681890833973; duration=0sec 2023-04-19 07:53:54,539 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:54,539 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/5e0ab40f3b86448b9c1cb49dc1a6d6fb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5e0ab40f3b86448b9c1cb49dc1a6d6fb 2023-04-19 07:53:54,550 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5e0ab40f3b86448b9c1cb49dc1a6d6fb, entries=150, sequenceid=250, filesize=11.8 K 2023-04-19 07:53:54,552 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/86729f33b4aa420d8243fc7aa85bf34b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/86729f33b4aa420d8243fc7aa85bf34b 2023-04-19 07:53:54,560 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/86729f33b4aa420d8243fc7aa85bf34b, entries=150, sequenceid=250, filesize=11.8 K 2023-04-19 07:53:54,561 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/7ae649aa61ae44759882e4512d2f51ee as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7ae649aa61ae44759882e4512d2f51ee 2023-04-19 07:53:54,573 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7ae649aa61ae44759882e4512d2f51ee, entries=150, sequenceid=250, filesize=11.8 K 2023-04-19 07:53:54,575 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 3c54cc0aa26fee1a2f14917923d1bb71 in 526ms, sequenceid=250, compaction requested=false 2023-04-19 07:53:54,575 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:54,575 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:54,576 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:54,576 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,576 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:54,576 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:54,581 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,581 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:54,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:54,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27a3ae5e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:54,585 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:54,585 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:54,588 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:54,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:54,588 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:54,588 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:54,588 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:54,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:54,590 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:54,590 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,590 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:54,590 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:54,594 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:54,594 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:54,594 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:54,594 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:54,594 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:54,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:54,596 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:54,596 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:54,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:54,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:54,597 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:54,597 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:54,599 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:54,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:54,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:54,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:54,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:54,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:54,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:54,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:54,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:54,602 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:54,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:54,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:54,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:54,603 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:54,603 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:54,703 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:54,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:54,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:54,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:54,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:54,709 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:54,709 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:54,710 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:54,710 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:54,710 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:54,710 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:54,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:54,712 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:54,714 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:54,714 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:54,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:54,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:54,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:54,715 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:54,715 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:54,715 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:54,716 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:54,716 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:54,716 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:54,716 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:54,716 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 07:53:54,717 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:54,717 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:54,717 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:54,717 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:54,717 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:54,717 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:54,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:54,792 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:54,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890894818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681890834806, max timestamp=1681890834818Num uniques 859; 2023-04-19 07:53:54,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-19 07:53:54,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890894818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681890834806, max timestamp=1681890834818Num uniques 859; 2023-04-19 07:53:54,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-19 07:53:54,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681890834806, max timestamp=1681890834818Num uniques 859; 2023-04-19 07:53:54,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-19 07:53:54,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57322 deadline: 1681890894874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890894923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890894924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:54,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:54,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57322 deadline: 1681890894984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:55,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890895128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:55,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890895129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,169 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/a777045aab7b4c758ce82b93c21a15ce 2023-04-19 07:53:55,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:55,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57322 deadline: 1681890895188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,199 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/dabe6f6e73414019b14156e8571a48fb 2023-04-19 07:53:55,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:55,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890895434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:55,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890895436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:55,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57322 deadline: 1681890895490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:55,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681890895544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,550 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8191 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:53:55,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:55,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57310 deadline: 1681890895551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,555 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8200 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:53:55,631 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/6f56d6f92b5e440c9113e348111b670f 2023-04-19 07:53:55,639 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/a777045aab7b4c758ce82b93c21a15ce as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/a777045aab7b4c758ce82b93c21a15ce 2023-04-19 07:53:55,646 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/a777045aab7b4c758ce82b93c21a15ce, entries=150, sequenceid=264, filesize=11.9 K 2023-04-19 07:53:55,647 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/dabe6f6e73414019b14156e8571a48fb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/dabe6f6e73414019b14156e8571a48fb 2023-04-19 07:53:55,654 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/dabe6f6e73414019b14156e8571a48fb, entries=150, sequenceid=264, filesize=11.9 K 2023-04-19 07:53:55,656 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/6f56d6f92b5e440c9113e348111b670f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/6f56d6f92b5e440c9113e348111b670f 2023-04-19 07:53:55,663 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/6f56d6f92b5e440c9113e348111b670f, entries=150, sequenceid=264, filesize=11.9 K 2023-04-19 07:53:55,664 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 3c54cc0aa26fee1a2f14917923d1bb71 in 948ms, sequenceid=264, compaction requested=true 2023-04-19 07:53:55,664 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:55,664 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:55,665 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:55,665 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,665 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:55,665 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:55,667 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,667 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:55,668 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:55,668 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:55,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:55,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c4824d2[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:55,670 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:55,670 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:55,671 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:55,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:55,672 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,672 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:55,674 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:55,674 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:55,674 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:55,675 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:55,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:55,678 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:55,678 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:55,678 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:55,678 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:55,678 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:55,681 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:55,682 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:55,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:55,682 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:55,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:55,682 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:55,684 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:55,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:55,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:55,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:55,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:55,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:55,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:55,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:55,689 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:55,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:55,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:55,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:55,690 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:55,690 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:55,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:55,790 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:55,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:55,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:55,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:55,795 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:55,796 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:55,796 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:55,796 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:55,796 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:55,796 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:55,797 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:55,797 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:55,799 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,799 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:55,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:55,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:55,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:55,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:55,801 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:55,801 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:55,801 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:55,801 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-04-19 07:53:55,802 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:55,802 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:55,802 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:55,802 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:55,802 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:55,802 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:55,814 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/b6701425e8aa40ba9225d91dae946824 2023-04-19 07:53:55,840 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/05d874b429d443568e5d95aaff0c91fa 2023-04-19 07:53:55,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:55,942 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:55,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:55,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890895958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:55,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:55,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890895962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:56,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57322 deadline: 1681890896000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:56,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890896063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:56,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890896066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:56,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890896268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:56,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890896270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,280 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/6ff7371e3a4c41ce827bd89c6e39054c 2023-04-19 07:53:56,291 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/b6701425e8aa40ba9225d91dae946824 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b6701425e8aa40ba9225d91dae946824 2023-04-19 07:53:56,305 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b6701425e8aa40ba9225d91dae946824, entries=150, sequenceid=297, filesize=11.9 K 2023-04-19 07:53:56,306 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/05d874b429d443568e5d95aaff0c91fa as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/05d874b429d443568e5d95aaff0c91fa 2023-04-19 07:53:56,315 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/05d874b429d443568e5d95aaff0c91fa, entries=150, sequenceid=297, filesize=11.9 K 2023-04-19 07:53:56,316 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/6ff7371e3a4c41ce827bd89c6e39054c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/6ff7371e3a4c41ce827bd89c6e39054c 2023-04-19 07:53:56,323 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/6ff7371e3a4c41ce827bd89c6e39054c, entries=150, sequenceid=297, filesize=11.9 K 2023-04-19 07:53:56,332 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=53.67 KB/54960 for 3c54cc0aa26fee1a2f14917923d1bb71 in 531ms, sequenceid=297, compaction requested=true 2023-04-19 07:53:56,332 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:56,333 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:56,333 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:56,334 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:56,335 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,335 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:56,335 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:56,405 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:56,406 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,406 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:56,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:56,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@757cf90a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:56,409 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:56,409 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,436 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:56,436 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,436 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:56,482 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:56,482 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:56,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:56,489 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:56,489 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:56,489 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:56,489 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:56,489 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:56,518 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,518 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:56,518 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:56,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:56,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,519 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,521 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,559 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,560 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:56,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:56,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:56,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:56,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:56,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:56,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:56,560 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:56,560 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:56,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:56,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:56,577 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 07:53:56,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:56,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:56,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:56,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:56,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:56,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:56,623 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/b095d5e158ca4202b2bcd164605d710e 2023-04-19 07:53:56,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:56,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890896634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:56,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890896635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,660 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:56,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:56,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:56,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:56,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:56,669 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:56,669 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:56,669 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:56,669 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:56,670 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,670 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,671 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/b9eb7b6b08bd4c8c9be5f32f12d9d7c8 2023-04-19 07:53:56,674 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:56,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:56,674 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:56,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:56,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:56,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:56,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:56,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:56,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:56,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:56,682 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:56,683 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,683 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:56,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:56,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:56,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:56,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:56,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:56,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71f0664f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:56,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:56,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,695 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:56,695 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,695 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:56,697 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:56,697 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:56,697 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:56,697 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:56,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:56,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:56,701 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:56,701 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:56,701 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:56,701 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:56,701 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:56,705 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,705 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:56,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,706 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:56,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,708 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,714 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,714 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,714 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:56,714 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:56,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:56,714 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:56,714 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,715 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,715 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,715 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,715 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:56,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:56,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:56,715 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:56,715 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:56,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:56,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:56,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890896737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:56,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890896740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,815 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:56,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:56,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:56,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:56,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:56,824 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:56,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:56,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:56,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:56,825 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,825 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,827 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,827 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:56,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:56,827 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,827 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:56,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:56,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:56,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:56,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:56,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:56,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:56,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:56,836 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:56,836 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:56,836 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:56,836 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:56,838 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,838 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,839 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,839 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b9bf771[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:56,853 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:56,853 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,855 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:56,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:56,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:56,857 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:56,857 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:56,857 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:56,858 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,858 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:56,858 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:56,859 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:56,859 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:56,859 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:56,859 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:56,859 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:56,866 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,866 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:56,866 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:56,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:56,867 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:56,869 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,873 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,873 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,874 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,873 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:56,874 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:56,874 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:56,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:56,874 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:56,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:56,874 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,874 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,874 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:56,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:56,875 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:56,875 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:56,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:56,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890896942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:56,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890896944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,975 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:56,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:56,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:56,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:56,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:56,982 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:56,982 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:56,982 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:56,982 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:56,982 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,982 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:56,984 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:56,987 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,987 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:56,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:56,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:56,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:56,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:56,989 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:56,989 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:56,989 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:56,989 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:56,990 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:56,991 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,991 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6382464[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:56,994 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:56,994 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,996 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:56,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:56,998 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,998 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:56,998 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:56,998 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:57,001 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:57,001 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:57,001 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:57,001 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:57,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:57,002 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:57,005 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,005 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:57,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,007 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:57,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57322 deadline: 1681890897009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,010 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,014 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:57,014 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:57,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:57,014 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,014 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,014 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,014 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,014 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,014 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,014 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:57,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:57,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:57,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:57,015 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:57,015 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:57,014 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:57,104 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/0d022e7c507f42b3b45351edd4216041 2023-04-19 07:53:57,111 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/b095d5e158ca4202b2bcd164605d710e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b095d5e158ca4202b2bcd164605d710e 2023-04-19 07:53:57,115 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:57,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:57,117 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b095d5e158ca4202b2bcd164605d710e, entries=200, sequenceid=309, filesize=14.3 K 2023-04-19 07:53:57,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:57,118 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/b9eb7b6b08bd4c8c9be5f32f12d9d7c8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/b9eb7b6b08bd4c8c9be5f32f12d9d7c8 2023-04-19 07:53:57,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:57,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:57,120 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:57,120 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:57,120 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:57,120 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:57,120 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,120 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,123 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,123 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:57,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:57,124 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,124 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/b9eb7b6b08bd4c8c9be5f32f12d9d7c8, entries=150, sequenceid=309, filesize=11.9 K 2023-04-19 07:53:57,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/0d022e7c507f42b3b45351edd4216041 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0d022e7c507f42b3b45351edd4216041 2023-04-19 07:53:57,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:57,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:57,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:57,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:57,126 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:57,127 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:57,127 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:57,131 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:57,131 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:57,131 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,131 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:57,131 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:57,131 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,131 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:57,131 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:57,131 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,131 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:57,131 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:57,133 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,133 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,133 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:57,133 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,133 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0d022e7c507f42b3b45351edd4216041, entries=150, sequenceid=309, filesize=11.9 K 2023-04-19 07:53:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,135 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3c54cc0aa26fee1a2f14917923d1bb71 in 558ms, sequenceid=309, compaction requested=true 2023-04-19 07:53:57,135 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:57,135 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:57,135 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:53:57,135 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:53:57,135 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:53:57,135 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:53:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:57,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@141e2900[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:57,138 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:57,138 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,139 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63446 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:53:57,139 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61006 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:53:57,139 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/A is initiating minor compaction (all files) 2023-04-19 07:53:57,139 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/B is initiating minor compaction (all files) 2023-04-19 07:53:57,139 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/A in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,139 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/B in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,139 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/7d2a9c4a00d040f9983c72d42add06ec, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5e0ab40f3b86448b9c1cb49dc1a6d6fb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/a777045aab7b4c758ce82b93c21a15ce, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b6701425e8aa40ba9225d91dae946824, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b095d5e158ca4202b2bcd164605d710e] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=62.0 K 2023-04-19 07:53:57,139 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/714a6ad28fc34152a6893c292b611f87, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/86729f33b4aa420d8243fc7aa85bf34b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/dabe6f6e73414019b14156e8571a48fb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/05d874b429d443568e5d95aaff0c91fa, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/b9eb7b6b08bd4c8c9be5f32f12d9d7c8] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=59.6 K 2023-04-19 07:53:57,139 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,141 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:57,142 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:57,142 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:57,142 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 7d2a9c4a00d040f9983c72d42add06ec, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681890833439 2023-04-19 07:53:57,142 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,142 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:57,142 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 714a6ad28fc34152a6893c292b611f87, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681890833439 2023-04-19 07:53:57,142 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 5e0ab40f3b86448b9c1cb49dc1a6d6fb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681890833534 2023-04-19 07:53:57,142 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 86729f33b4aa420d8243fc7aa85bf34b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681890833534 2023-04-19 07:53:57,143 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting a777045aab7b4c758ce82b93c21a15ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681890834169 2023-04-19 07:53:57,143 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting dabe6f6e73414019b14156e8571a48fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681890834169 2023-04-19 07:53:57,143 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting b6701425e8aa40ba9225d91dae946824, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1681890834842 2023-04-19 07:53:57,143 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 05d874b429d443568e5d95aaff0c91fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1681890834842 2023-04-19 07:53:57,144 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting b095d5e158ca4202b2bcd164605d710e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681890835956 2023-04-19 07:53:57,144 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting b9eb7b6b08bd4c8c9be5f32f12d9d7c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681890835956 2023-04-19 07:53:57,146 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:57,146 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:57,147 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:57,147 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:57,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:57,152 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:57,152 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:57,152 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:57,152 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:57,152 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:57,154 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,154 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,154 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,157 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,162 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#B#compaction#48 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:57,163 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,166 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,166 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:57,166 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:57,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:57,166 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,166 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:57,166 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,166 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:57,166 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,167 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,167 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:57,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:57,167 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:57,167 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:57,167 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#A#compaction#49 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:57,204 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/daad1104dac14329b92b6608fb683217 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/daad1104dac14329b92b6608fb683217 2023-04-19 07:53:57,212 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/B of 3c54cc0aa26fee1a2f14917923d1bb71 into daad1104dac14329b92b6608fb683217(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:53:57,212 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:57,212 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/B, priority=11, startTime=1681890837135; duration=0sec 2023-04-19 07:53:57,212 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:53:57,212 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:53:57,215 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61006 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:53:57,216 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/C is initiating minor compaction (all files) 2023-04-19 07:53:57,216 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/C in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,216 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/5e4750d2087c49ce9f930d341d0f54fc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7ae649aa61ae44759882e4512d2f51ee, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/6f56d6f92b5e440c9113e348111b670f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/6ff7371e3a4c41ce827bd89c6e39054c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0d022e7c507f42b3b45351edd4216041] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=59.6 K 2023-04-19 07:53:57,216 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 5e4750d2087c49ce9f930d341d0f54fc, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681890833439 2023-04-19 07:53:57,217 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7ae649aa61ae44759882e4512d2f51ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681890833534 2023-04-19 07:53:57,218 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 6f56d6f92b5e440c9113e348111b670f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681890834169 2023-04-19 07:53:57,218 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 6ff7371e3a4c41ce827bd89c6e39054c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1681890834842 2023-04-19 07:53:57,219 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 0d022e7c507f42b3b45351edd4216041, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681890835956 2023-04-19 07:53:57,232 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#C#compaction#50 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:57,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:57,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890836625, max timestamp=1681890837246Num uniques 859; 2023-04-19 07:53:57,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890836625, max timestamp=1681890837246Num uniques 859; 2023-04-19 07:53:57,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:53:57,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890836625, max timestamp=1681890837246Num uniques 859; 2023-04-19 07:53:57,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:53:57,248 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-19 07:53:57,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:53:57,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:57,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:57,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:57,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:57,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:57,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:57,260 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/d16492b35c5f4ae4b5fde196ef781775 2023-04-19 07:53:57,267 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:57,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:57,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:57,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:57,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:57,271 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:57,271 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:57,271 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:57,271 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:57,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,273 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:57,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:57,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:57,273 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,274 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,274 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,274 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/214cf6a0cd7b40fd9ba890457435ccbc 2023-04-19 07:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:57,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:57,275 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:57,275 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:57,275 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:57,276 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,276 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:57,276 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:57,276 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:57,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:57,280 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,280 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,280 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,280 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66aef7d2[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:57,284 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:57,284 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,285 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,286 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,286 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:57,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:57,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:57,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:57,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,287 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:57,287 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:57,287 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:57,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:57,297 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:57,297 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:57,297 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:57,297 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:57,297 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:57,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890837260, max timestamp=1681890837296Num uniques 859; 2023-04-19 07:53:57,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890837260, max timestamp=1681890837296Num uniques 859; 2023-04-19 07:53:57,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:53:57,299 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:53:57,300 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,300 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,300 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890837260, max timestamp=1681890837296Num uniques 859; 2023-04-19 07:53:57,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:53:57,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,301 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,303 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,308 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,308 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:57,308 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:57,308 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:57,308 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:57,308 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:57,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:57,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:57,308 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,309 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,309 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:57,309 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:57,309 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,309 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:57,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/9590d2647cea429dabc5b06d48f5af85 2023-04-19 07:53:57,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/d16492b35c5f4ae4b5fde196ef781775 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/d16492b35c5f4ae4b5fde196ef781775 2023-04-19 07:53:57,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/d16492b35c5f4ae4b5fde196ef781775, entries=200, sequenceid=331, filesize=14.3 K 2023-04-19 07:53:57,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/214cf6a0cd7b40fd9ba890457435ccbc as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/214cf6a0cd7b40fd9ba890457435ccbc 2023-04-19 07:53:57,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/214cf6a0cd7b40fd9ba890457435ccbc, entries=150, sequenceid=331, filesize=11.9 K 2023-04-19 07:53:57,342 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/9590d2647cea429dabc5b06d48f5af85 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/9590d2647cea429dabc5b06d48f5af85 2023-04-19 07:53:57,353 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/9590d2647cea429dabc5b06d48f5af85, entries=150, sequenceid=331, filesize=11.9 K 2023-04-19 07:53:57,354 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=118.53 KB/121370 for 3c54cc0aa26fee1a2f14917923d1bb71 in 106ms, sequenceid=331, compaction requested=false 2023-04-19 07:53:57,354 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:57,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:57,357 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-04-19 07:53:57,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:57,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:57,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:57,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:57,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:57,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:57,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/ab05e2b84e724a51989e7f9f9e168b9a 2023-04-19 07:53:57,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:57,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890897381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:57,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890897384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,406 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/6e952b43ae774b759a0bcdc6c7499213 2023-04-19 07:53:57,409 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:57,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:57,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:57,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:57,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:57,416 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:57,416 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:57,416 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:57,416 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:57,417 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,418 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,419 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:57,419 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:57,419 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,420 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:57,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:57,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:57,424 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:57,424 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:57,424 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:57,428 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,428 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:57,428 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:57,428 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:57,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:57,430 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,430 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,430 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,430 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@721aa409[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:57,432 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:57,432 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,434 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,434 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:57,434 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:57,434 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:57,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:57,439 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:57,441 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:57,441 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:57,441 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:57,441 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:57,441 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:57,443 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,443 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:57,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,443 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,444 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,446 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,449 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:57,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:57,449 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,449 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:57,449 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:57,449 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,449 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,449 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,449 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,449 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,449 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:57,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:57,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:57,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:57,450 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:57,450 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:57,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:57,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890897485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:57,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890897504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,550 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:57,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:57,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:57,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:57,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:57,554 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:57,554 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:57,554 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:57,554 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:57,554 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,555 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,556 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,556 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:57,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:57,556 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,556 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:57,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:57,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:57,558 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,558 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:57,558 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:57,558 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:57,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:57,561 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,561 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,562 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,562 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:57,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3546a52a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:57,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:57,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,565 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:57,565 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:57,565 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,565 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:57,567 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:57,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,567 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,567 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:57,569 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:57,569 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:57,569 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:57,569 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:57,569 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:57,571 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,571 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,571 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,571 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,580 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,580 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:57,580 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:57,580 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,580 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,580 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:57,580 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,580 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:57,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:57,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:57,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:57,580 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:57,581 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:57,581 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:57,581 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,597 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/27c42c3122ee45edaebdd9a21676c3f2 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/27c42c3122ee45edaebdd9a21676c3f2 2023-04-19 07:53:57,604 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/A of 3c54cc0aa26fee1a2f14917923d1bb71 into 27c42c3122ee45edaebdd9a21676c3f2(size=12.4 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:53:57,604 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:57,604 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/A, priority=11, startTime=1681890837135; duration=0sec 2023-04-19 07:53:57,604 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:57,645 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/5bc9d21f07ec44298747cfd87b91e294 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/5bc9d21f07ec44298747cfd87b91e294 2023-04-19 07:53:57,652 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/C of 3c54cc0aa26fee1a2f14917923d1bb71 into 5bc9d21f07ec44298747cfd87b91e294(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:53:57,653 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:57,653 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/C, priority=11, startTime=1681890837135; duration=0sec 2023-04-19 07:53:57,653 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:57,681 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:57,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:57,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:57,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:57,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:57,688 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:57,688 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:57,688 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:57,688 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:57,689 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,689 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,690 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:57,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:57,690 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,691 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,691 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:57,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:57,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890897689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:57,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:57,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:57,699 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:57,699 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:57,699 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:57,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:57,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:57,700 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,700 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:57,700 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:57,700 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:57,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:57,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:57,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:57,702 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,702 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,702 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,702 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:57,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50f5ccab[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:57,704 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:57,704 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,705 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,705 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:57,706 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:57,706 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:57,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:57,708 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:57,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:57,710 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:57,710 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:57,710 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:57,710 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:57,710 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:57,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:57,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890897710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:57,713 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,715 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,717 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,717 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:57,717 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:57,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:57,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:57,717 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:57,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:57,717 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:57,717 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,717 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:57,718 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:57,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:57,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,818 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:57,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:57,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:57,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:57,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:57,822 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:57,822 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:57,822 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:57,822 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:57,823 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,823 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,824 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:57,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:57,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,825 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,825 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:57,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:57,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:57,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:57,844 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:57,844 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:57,844 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/e15b4cd378724f13b4f9d14e51e69803 2023-04-19 07:53:57,853 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/ab05e2b84e724a51989e7f9f9e168b9a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/ab05e2b84e724a51989e7f9f9e168b9a 2023-04-19 07:53:57,856 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,856 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:57,856 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:57,856 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:57,860 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:57,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41f4b100[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:57,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:57,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,864 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,864 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/ab05e2b84e724a51989e7f9f9e168b9a, entries=300, sequenceid=367, filesize=19.0 K 2023-04-19 07:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:57,865 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,868 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:57,869 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/6e952b43ae774b759a0bcdc6c7499213 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6e952b43ae774b759a0bcdc6c7499213 2023-04-19 07:53:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:57,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:57,880 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:57,880 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:57,880 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:57,880 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:57,880 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:57,887 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,887 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:57,887 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:57,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:57,890 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6e952b43ae774b759a0bcdc6c7499213, entries=150, sequenceid=367, filesize=11.9 K 2023-04-19 07:53:57,890 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,891 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/e15b4cd378724f13b4f9d14e51e69803 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/e15b4cd378724f13b4f9d14e51e69803 2023-04-19 07:53:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,902 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:57,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,907 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/e15b4cd378724f13b4f9d14e51e69803, entries=150, sequenceid=367, filesize=11.9 K 2023-04-19 07:53:57,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:57,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:57,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:57,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:57,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:57,908 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:57,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:57,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:57,909 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,909 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:57,909 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:57,909 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:57,909 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:57,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:57,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:57,909 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=46.96 KB/48090 for 3c54cc0aa26fee1a2f14917923d1bb71 in 552ms, sequenceid=367, compaction requested=true 2023-04-19 07:53:57,910 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:57,910 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:57,910 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:53:57,910 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:53:57,910 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:57,910 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:53:57,912 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 46864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:53:57,912 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:53:57,912 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/A is initiating minor compaction (all files) 2023-04-19 07:53:57,912 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/B is initiating minor compaction (all files) 2023-04-19 07:53:57,912 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/A in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,912 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/B in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:57,912 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/27c42c3122ee45edaebdd9a21676c3f2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/d16492b35c5f4ae4b5fde196ef781775, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/ab05e2b84e724a51989e7f9f9e168b9a] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=45.8 K 2023-04-19 07:53:57,912 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/daad1104dac14329b92b6608fb683217, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/214cf6a0cd7b40fd9ba890457435ccbc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6e952b43ae774b759a0bcdc6c7499213] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=36.2 K 2023-04-19 07:53:57,916 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 27c42c3122ee45edaebdd9a21676c3f2, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681890835956 2023-04-19 07:53:57,917 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting daad1104dac14329b92b6608fb683217, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681890835956 2023-04-19 07:53:57,933 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 214cf6a0cd7b40fd9ba890457435ccbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681890836625 2023-04-19 07:53:57,933 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting d16492b35c5f4ae4b5fde196ef781775, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681890836625 2023-04-19 07:53:57,934 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 6e952b43ae774b759a0bcdc6c7499213, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681890837349 2023-04-19 07:53:57,934 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting ab05e2b84e724a51989e7f9f9e168b9a, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681890837316 2023-04-19 07:53:57,953 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#B#compaction#57 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:57,956 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#A#compaction#58 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:57,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:57,995 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 07:53:57,996 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:57,996 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:57,996 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:57,996 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:57,996 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:57,996 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:58,008 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/64ae6381c9994c91b7001d5e1f7023e2 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/64ae6381c9994c91b7001d5e1f7023e2 2023-04-19 07:53:58,009 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:58,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:58,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:58,021 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/B of 3c54cc0aa26fee1a2f14917923d1bb71 into 64ae6381c9994c91b7001d5e1f7023e2(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:53:58,021 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:58,021 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/B, priority=13, startTime=1681890837910; duration=0sec 2023-04-19 07:53:58,021 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/8c55ea119cfb49f789effc59c8d4bcb9 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/8c55ea119cfb49f789effc59c8d4bcb9 2023-04-19 07:53:58,021 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:53:58,021 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:53:58,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:58,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:58,023 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:58,023 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:58,023 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:58,023 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:58,023 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,024 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,025 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:58,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:58,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:58,025 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,026 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:53:58,026 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/C is initiating minor compaction (all files) 2023-04-19 07:53:58,027 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/C in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:58,027 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/5bc9d21f07ec44298747cfd87b91e294, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/9590d2647cea429dabc5b06d48f5af85, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/e15b4cd378724f13b4f9d14e51e69803] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=36.2 K 2023-04-19 07:53:58,027 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 5bc9d21f07ec44298747cfd87b91e294, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681890835956 2023-04-19 07:53:58,028 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 9590d2647cea429dabc5b06d48f5af85, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681890836625 2023-04-19 07:53:58,029 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting e15b4cd378724f13b4f9d14e51e69803, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681890837349 2023-04-19 07:53:58,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,031 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,031 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:58,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:58,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:58,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:58,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:58,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:58,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890838042, max timestamp=1681890838051Num uniques 859; 2023-04-19 07:53:58,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:53:58,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890838042, max timestamp=1681890838051Num uniques 859; 2023-04-19 07:53:58,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890838042, max timestamp=1681890838051Num uniques 859; 2023-04-19 07:53:58,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:53:58,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:53:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:58,068 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:58,068 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:58,068 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:58,068 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:58,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,070 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#C#compaction#60 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:58,072 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:58,072 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:58,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,074 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:58,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:58,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:58,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66e96fdb[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:58,080 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:58,080 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,080 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/A of 3c54cc0aa26fee1a2f14917923d1bb71 into 8c55ea119cfb49f789effc59c8d4bcb9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:53:58,080 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:58,080 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/A, priority=13, startTime=1681890837910; duration=0sec 2023-04-19 07:53:58,080 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:58,081 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,082 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:58,082 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:58,082 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:58,082 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,082 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:58,085 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:58,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:58,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:58,108 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:58,108 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:58,108 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:58,108 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:58,108 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:58,110 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,110 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:58,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:58,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:58,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890898105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,110 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:58,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:58,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:58,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:58,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890898107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:58,114 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,129 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:58,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,134 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:58,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:58,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:58,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:58,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:58,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:58,135 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:58,135 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:58,135 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:58,135 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,135 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:58,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:58,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890898216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:58,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890898226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,235 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:58,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:58,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:58,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:58,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:58,240 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:58,240 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:58,240 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:58,240 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:58,241 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,241 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,243 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,243 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:58,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:58,245 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,245 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:58,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:58,246 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:58,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:58,246 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:58,246 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:58,246 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:58,248 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:58,248 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:58,248 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:58,248 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:58,250 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,250 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,250 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,250 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:58,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:58,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:58,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6606f9d9[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:58,253 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:58,253 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,254 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,254 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:58,254 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,255 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:58,254 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:58,255 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:58,257 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:58,257 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:58,257 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:58,257 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:58,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:58,260 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:58,260 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:58,260 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:58,260 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:58,260 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:58,262 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,262 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:58,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,262 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:58,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:58,266 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:58,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,267 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:58,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:58,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,270 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:58,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:58,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:58,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:58,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:58,271 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:58,271 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:58,371 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:58,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:58,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:58,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:58,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:58,377 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:58,377 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:58,377 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:58,377 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:58,379 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,379 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,380 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,381 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:58,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,382 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,382 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:58,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:58,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:58,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:58,383 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:58,383 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:58,383 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:58,383 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:58,384 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:58,384 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,384 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:58,384 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:58,385 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,386 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:58,387 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,387 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:58,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:58,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1868e126[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:58,391 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:58,391 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,393 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:58,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:58,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:58,393 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,393 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:58,395 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:58,395 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,395 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:58,395 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:58,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:58,397 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:58,398 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:58,398 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:58,398 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:58,398 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:58,400 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,400 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:58,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,400 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:58,401 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:58,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:58,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:58,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,402 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,406 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:58,406 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:58,406 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:58,406 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:58,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:58,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:58,406 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,407 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,407 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,407 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,407 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,407 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:58,407 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:58,407 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:58,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:58,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:58,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:58,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890898423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:58,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890898449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,479 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/dc42a823b32b4397ba01bd0d1d3effe9 2023-04-19 07:53:58,488 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/aa005343b54a47b884ef5242840c9cb5 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/aa005343b54a47b884ef5242840c9cb5 2023-04-19 07:53:58,496 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/C of 3c54cc0aa26fee1a2f14917923d1bb71 into aa005343b54a47b884ef5242840c9cb5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:53:58,496 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:58,496 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/C, priority=13, startTime=1681890837910; duration=0sec 2023-04-19 07:53:58,496 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:58,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/ad99d35691df4158b37872f90bec0406 2023-04-19 07:53:58,507 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:58,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:58,509 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/7d2fdb20ed8948d6b6958bd97ae96572 2023-04-19 07:53:58,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:58,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:58,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:58,514 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:58,514 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:58,514 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:58,514 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:58,514 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/dc42a823b32b4397ba01bd0d1d3effe9 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/dc42a823b32b4397ba01bd0d1d3effe9 2023-04-19 07:53:58,515 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,515 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,517 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,517 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:58,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:58,517 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,517 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,519 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/dc42a823b32b4397ba01bd0d1d3effe9, entries=150, sequenceid=380, filesize=11.9 K 2023-04-19 07:53:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:58,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/ad99d35691df4158b37872f90bec0406 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/ad99d35691df4158b37872f90bec0406 2023-04-19 07:53:58,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:58,521 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:58,521 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:58,521 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:58,521 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:58,521 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:58,521 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:58,521 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:58,521 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:58,521 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:58,522 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:58,522 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:58,522 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,522 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:58,522 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:58,523 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,523 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:58,524 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,524 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:58,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:58,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cf02f9a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:58,526 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:58,526 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,527 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:58,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:58,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:58,528 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,528 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:58,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/ad99d35691df4158b37872f90bec0406, entries=150, sequenceid=380, filesize=11.9 K 2023-04-19 07:53:58,529 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:58,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:58,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:58,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/7d2fdb20ed8948d6b6958bd97ae96572 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7d2fdb20ed8948d6b6958bd97ae96572 2023-04-19 07:53:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:58,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:58,532 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:58,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:58,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:58,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:58,533 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:58,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7d2fdb20ed8948d6b6958bd97ae96572, entries=150, sequenceid=380, filesize=11.9 K 2023-04-19 07:53:58,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:58,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,536 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 3c54cc0aa26fee1a2f14917923d1bb71 in 541ms, sequenceid=380, compaction requested=false 2023-04-19 07:53:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:58,536 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:58,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,537 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,538 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:58,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:58,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:58,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:58,541 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:58,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:58,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:58,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:58,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,542 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,542 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:58,542 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:58,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:58,542 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:58,542 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:58,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:58,642 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:58,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:58,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:58,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:58,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:58,647 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:58,647 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:58,647 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:58,647 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:58,648 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,649 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,656 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:58,656 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:58,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:58,657 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,657 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:58,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:58,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:58,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:58,658 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:58,658 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:58,658 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:58,666 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:58,666 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:58,666 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:58,666 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:58,666 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-19 07:53:58,666 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:58,666 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:58,667 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:58,667 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:58,667 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:58,667 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:58,702 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/3cda03775dc0482598c8500181356856 2023-04-19 07:53:58,720 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/04dfb2fabbfd4d8c82c5f17bc6b76d5e 2023-04-19 07:53:58,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:58,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:58,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:58,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890898751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:58,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890898754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:58,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:58,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890898854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:59,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57322 deadline: 1681890899016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,034 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:53:59,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:59,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890899058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,147 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/b60b5afe680d40c791565524274d5df6 2023-04-19 07:53:59,154 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/3cda03775dc0482598c8500181356856 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/3cda03775dc0482598c8500181356856 2023-04-19 07:53:59,160 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/3cda03775dc0482598c8500181356856, entries=150, sequenceid=418, filesize=11.9 K 2023-04-19 07:53:59,161 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/04dfb2fabbfd4d8c82c5f17bc6b76d5e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/04dfb2fabbfd4d8c82c5f17bc6b76d5e 2023-04-19 07:53:59,167 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/04dfb2fabbfd4d8c82c5f17bc6b76d5e, entries=150, sequenceid=418, filesize=11.9 K 2023-04-19 07:53:59,168 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/b60b5afe680d40c791565524274d5df6 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b60b5afe680d40c791565524274d5df6 2023-04-19 07:53:59,173 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b60b5afe680d40c791565524274d5df6, entries=150, sequenceid=418, filesize=11.9 K 2023-04-19 07:53:59,174 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 3c54cc0aa26fee1a2f14917923d1bb71 in 508ms, sequenceid=418, compaction requested=true 2023-04-19 07:53:59,174 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:59,174 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:59,174 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:59,174 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:59,174 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,176 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:59,176 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:59,177 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,177 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,178 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,178 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:59,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@514a1068[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:59,181 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:59,181 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,183 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,183 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:59,183 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:59,183 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:59,183 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,183 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:59,185 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:59,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:59,188 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:59,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:59,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:59,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:59,188 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:59,190 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,190 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:59,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,190 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:59,191 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,192 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,194 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,195 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:59,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:59,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:59,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:59,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:59,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:59,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:59,195 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:59,195 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:59,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:59,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:59,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:59,263 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 07:53:59,263 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:59,263 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:59,263 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:59,263 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:59,263 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:59,263 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:53:59,295 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:59,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:59,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:59,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:59,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:59,305 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:59,305 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:59,305 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:59,305 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:59,306 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,306 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,309 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,309 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:59,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:59,309 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,309 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:59,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:59,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:59,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:59,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:59,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:59,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:59,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:59,311 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:59,311 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:59,311 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:59,311 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:59,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:59,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:59,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:59,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:59,313 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,314 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,314 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,315 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:59,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53f3a9[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:59,320 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:59,320 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,322 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:59,322 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:59,322 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:59,322 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,322 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:59,323 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:59,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:59,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:59,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:59,327 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:59,327 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:59,327 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:59,327 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:59,327 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:59,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:59,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890899327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,330 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,330 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,330 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,331 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,344 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:59,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:59,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:59,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:59,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:59,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:59,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,349 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,349 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,349 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,349 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:59,349 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:59,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:59,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:59,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:59,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890899362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:59,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890899429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,449 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:59,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:59,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:59,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:59,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:59,453 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:59,453 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:59,453 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:59,453 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:59,453 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,453 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,455 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,455 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:59,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:59,455 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,455 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:59,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:59,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:59,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:59,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:59,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:59,457 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:59,457 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:59,457 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:59,457 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:59,457 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:59,457 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:59,457 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:59,457 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:59,457 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,457 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:59,457 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:59,459 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,459 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,459 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,459 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:59,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f6d4354[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:59,461 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:59,461 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,462 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:59,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:59,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:59,463 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,463 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:59,466 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:59,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:59,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:59,469 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:59,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:59,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:59,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:59,469 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:59,470 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,470 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,470 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,472 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,477 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:59,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:59,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:59,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:59,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:59,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,480 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:59,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,480 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:59,480 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:59,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,481 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,481 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,481 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:59,581 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:59,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:59,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:59,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:59,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:59,585 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:59,585 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:59,585 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:59,585 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:59,585 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,585 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,587 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,587 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:59,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:59,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:59,587 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,587 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:59,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:59,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:59,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:59,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:59,589 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:59,589 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:59,589 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:59,589 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:59,589 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:59,589 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:59,589 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:59,589 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:59,589 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:59,590 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:59,590 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:59,590 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,590 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:59,590 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:59,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:59,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@229c3bdc[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:59,595 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:59,595 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,603 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:59,604 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:59,604 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:59,605 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,605 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,605 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:59,605 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:59,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:59,608 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:59,608 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:59,608 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:59,608 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:59,608 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:59,610 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,610 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,610 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:59,611 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,612 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,614 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,614 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:59,614 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:59,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:59,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:59,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:59,615 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:59,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,615 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:59,615 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:59,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:59,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:53:59,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890899632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,678 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/5b42b0b64928408ba7bd001bd709f293 2023-04-19 07:53:59,715 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:59,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:59,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/f92964c2c94e4440a3ba6f319df52a7a 2023-04-19 07:53:59,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:59,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:59,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:59,719 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:59,719 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:59,719 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:59,719 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:59,720 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,720 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,721 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:59,721 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:59,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,722 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,722 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:59,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:59,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:59,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:59,723 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:59,723 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:59,723 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:59,723 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:59,723 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:59,723 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:59,723 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:59,723 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:59,723 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:59,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:59,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:59,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:59,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:59,730 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,731 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,732 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,732 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:59,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:59,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23d4a6b7[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:59,736 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:59,736 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,739 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,739 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:59,740 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,741 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:59,741 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:59,741 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:59,743 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/54e0407fd32144eca85f715990e0501c 2023-04-19 07:53:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:59,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:59,748 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:59,748 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:59,748 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:59,748 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:59,748 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:59,751 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,751 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,751 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:59,751 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/5b42b0b64928408ba7bd001bd709f293 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5b42b0b64928408ba7bd001bd709f293 2023-04-19 07:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,753 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,756 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,761 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5b42b0b64928408ba7bd001bd709f293, entries=150, sequenceid=430, filesize=11.9 K 2023-04-19 07:53:59,762 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/f92964c2c94e4440a3ba6f319df52a7a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/f92964c2c94e4440a3ba6f319df52a7a 2023-04-19 07:53:59,765 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:59,765 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:59,765 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,765 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,765 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,765 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:59,765 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:59,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:59,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:59,765 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,766 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,766 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,766 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:59,766 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:59,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:59,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/f92964c2c94e4440a3ba6f319df52a7a, entries=150, sequenceid=430, filesize=11.9 K 2023-04-19 07:53:59,771 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/54e0407fd32144eca85f715990e0501c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/54e0407fd32144eca85f715990e0501c 2023-04-19 07:53:59,779 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/54e0407fd32144eca85f715990e0501c, entries=150, sequenceid=430, filesize=11.9 K 2023-04-19 07:53:59,780 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3c54cc0aa26fee1a2f14917923d1bb71 in 517ms, sequenceid=430, compaction requested=true 2023-04-19 07:53:59,780 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:59,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:59,780 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:53:59,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:53:59,780 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:53:59,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:53:59,782 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:53:59,782 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:53:59,782 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/B is initiating minor compaction (all files) 2023-04-19 07:53:59,782 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/A is initiating minor compaction (all files) 2023-04-19 07:53:59,782 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/B in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:59,782 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/A in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:59,783 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/64ae6381c9994c91b7001d5e1f7023e2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/ad99d35691df4158b37872f90bec0406, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/04dfb2fabbfd4d8c82c5f17bc6b76d5e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/f92964c2c94e4440a3ba6f319df52a7a] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=48.2 K 2023-04-19 07:53:59,783 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/8c55ea119cfb49f789effc59c8d4bcb9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/dc42a823b32b4397ba01bd0d1d3effe9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/3cda03775dc0482598c8500181356856, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5b42b0b64928408ba7bd001bd709f293] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=48.2 K 2023-04-19 07:53:59,783 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 64ae6381c9994c91b7001d5e1f7023e2, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681890837349 2023-04-19 07:53:59,784 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 8c55ea119cfb49f789effc59c8d4bcb9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681890837349 2023-04-19 07:53:59,784 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting ad99d35691df4158b37872f90bec0406, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681890837365 2023-04-19 07:53:59,784 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting dc42a823b32b4397ba01bd0d1d3effe9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681890837365 2023-04-19 07:53:59,784 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 04dfb2fabbfd4d8c82c5f17bc6b76d5e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1681890838099 2023-04-19 07:53:59,785 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 3cda03775dc0482598c8500181356856, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1681890838099 2023-04-19 07:53:59,785 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting f92964c2c94e4440a3ba6f319df52a7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681890838741 2023-04-19 07:53:59,785 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 5b42b0b64928408ba7bd001bd709f293, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681890838741 2023-04-19 07:53:59,805 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#A#compaction#69 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:59,806 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#B#compaction#70 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:59,835 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/38f991b196e347aab26c76e9aa024496 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/38f991b196e347aab26c76e9aa024496 2023-04-19 07:53:59,846 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/B of 3c54cc0aa26fee1a2f14917923d1bb71 into 38f991b196e347aab26c76e9aa024496(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:53:59,846 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:59,846 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/B, priority=12, startTime=1681890839780; duration=0sec 2023-04-19 07:53:59,846 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:53:59,846 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:53:59,847 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:53:59,847 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/C is initiating minor compaction (all files) 2023-04-19 07:53:59,848 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/C in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:59,848 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/aa005343b54a47b884ef5242840c9cb5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7d2fdb20ed8948d6b6958bd97ae96572, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b60b5afe680d40c791565524274d5df6, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/54e0407fd32144eca85f715990e0501c] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=48.2 K 2023-04-19 07:53:59,848 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting aa005343b54a47b884ef5242840c9cb5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681890837349 2023-04-19 07:53:59,849 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7d2fdb20ed8948d6b6958bd97ae96572, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681890837365 2023-04-19 07:53:59,849 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting b60b5afe680d40c791565524274d5df6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1681890838099 2023-04-19 07:53:59,850 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 54e0407fd32144eca85f715990e0501c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681890838741 2023-04-19 07:53:59,862 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#C#compaction#71 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:53:59,866 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:53:59,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:53:59,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:53:59,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890839307, max timestamp=1681890839870Num uniques 859; 2023-04-19 07:53:59,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:53:59,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890839307, max timestamp=1681890839870Num uniques 859; 2023-04-19 07:53:59,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:53:59,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890839307, max timestamp=1681890839870Num uniques 859; 2023-04-19 07:53:59,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:53:59,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:53:59,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:53:59,880 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:53:59,880 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:59,880 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:53:59,880 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:53:59,882 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,882 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,883 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:59,884 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:59,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:59,884 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,884 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:53:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:53:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:53:59,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:53:59,893 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-19 07:53:59,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:53:59,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:59,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:53:59,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:59,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:53:59,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:53:59,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:53:59,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:53:59,896 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:53:59,896 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:53:59,896 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:53:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:53:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:53:59,900 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:59,900 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:53:59,900 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:53:59,900 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:53:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:53:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:53:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:53:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:53:59,903 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:53:59,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c85c7df[Count = 0] remaining members to acquire global barrier 2023-04-19 07:53:59,905 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:53:59,905 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,907 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,907 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:53:59,909 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:53:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:53:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:53:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:53:59,912 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,912 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:53:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:59,913 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:53:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,917 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/19096e4b768f4ad6aea590da6c7c0698 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/19096e4b768f4ad6aea590da6c7c0698 2023-04-19 07:53:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:53:59,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:53:59,918 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:53:59,918 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:53:59,918 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:53:59,918 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:53:59,918 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:53:59,920 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,920 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:59,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,921 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:53:59,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:59,921 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:53:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:53:59,923 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:53:59,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:53:59,926 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/C of 3c54cc0aa26fee1a2f14917923d1bb71 into 19096e4b768f4ad6aea590da6c7c0698(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:53:59,926 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:53:59,926 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/C, priority=12, startTime=1681890839780; duration=0sec 2023-04-19 07:53:59,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:53:59,927 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:53:59,927 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/a45df8dd9d2b4b12a2bbbd08fa7258d7 2023-04-19 07:53:59,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:53:59,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:53:59,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:53:59,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:53:59,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:53:59,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:53:59,927 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:53:59,928 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:53:59,927 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:53:59,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:53:59,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:53:59,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:53:59,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:53:59,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:53:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:53:59,988 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/aed165a0573448a1853f545edeb74e9c 2023-04-19 07:53:59,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890839956, max timestamp=1681890839992Num uniques 859; 2023-04-19 07:53:59,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:53:59,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890839956, max timestamp=1681890839992Num uniques 859; 2023-04-19 07:53:59,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:53:59,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890839956, max timestamp=1681890839992Num uniques 859; 2023-04-19 07:53:59,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:00,028 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:00,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:00,031 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/485fb7eba7bd4a79bad392b08d21c24a 2023-04-19 07:54:00,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:00,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:00,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:00,032 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:00,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:00,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:00,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:00,033 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,033 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,034 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:00,034 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:00,035 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,035 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,038 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/a45df8dd9d2b4b12a2bbbd08fa7258d7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/a45df8dd9d2b4b12a2bbbd08fa7258d7 2023-04-19 07:54:00,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:00,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/a45df8dd9d2b4b12a2bbbd08fa7258d7, entries=200, sequenceid=456, filesize=14.3 K 2023-04-19 07:54:00,046 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/aed165a0573448a1853f545edeb74e9c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/aed165a0573448a1853f545edeb74e9c 2023-04-19 07:54:00,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:00,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:00,047 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:00,047 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:00,047 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:00,053 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/aed165a0573448a1853f545edeb74e9c, entries=150, sequenceid=456, filesize=11.9 K 2023-04-19 07:54:00,054 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/485fb7eba7bd4a79bad392b08d21c24a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/485fb7eba7bd4a79bad392b08d21c24a 2023-04-19 07:54:00,056 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:00,056 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:00,056 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:00,056 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:54:00,056 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:54:00,056 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:00,056 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:00,056 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:00,056 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,060 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/485fb7eba7bd4a79bad392b08d21c24a, entries=150, sequenceid=456, filesize=11.9 K 2023-04-19 07:54:00,061 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=127.47 KB/130530 for 3c54cc0aa26fee1a2f14917923d1bb71 in 168ms, sequenceid=456, compaction requested=false 2023-04-19 07:54:00,061 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:00,062 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:00,063 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:00,066 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,066 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:00,069 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,069 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:00,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:00,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9c754f8[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:00,072 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:00,073 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,074 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:00,074 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:00,074 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:00,075 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,075 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:00,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:54:00,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681890840035, max timestamp=1681890840074Num uniques 1018; 2023-04-19 07:54:00,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681890840035, max timestamp=1681890840074Num uniques 1018; 2023-04-19 07:54:00,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-19 07:54:00,079 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=69.33 KB heapSize=188.95 KB 2023-04-19 07:54:00,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-19 07:54:00,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681890840035, max timestamp=1681890840074Num uniques 1018; 2023-04-19 07:54:00,080 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-19 07:54:00,080 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:54:00,080 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,080 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:00,081 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:00,081 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:54:00,081 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:00,081 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:00,081 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:00,081 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:54:00,081 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:00,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:00,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:00,091 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:00,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:00,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:00,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:00,091 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:00,095 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:00,095 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,095 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:00,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,096 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:00,098 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,101 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:00,101 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:00,101 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:00,101 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:00,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:00,101 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,101 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,101 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,101 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,102 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,101 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:00,102 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:00,102 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:00,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:00,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890840111, max timestamp=1681890840122Num uniques 859; 2023-04-19 07:54:00,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:54:00,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890840111, max timestamp=1681890840122Num uniques 859; 2023-04-19 07:54:00,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:54:00,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890840111, max timestamp=1681890840122Num uniques 859; 2023-04-19 07:54:00,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:00,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681890840170, max timestamp=1681890840190Num uniques 1068; 2023-04-19 07:54:00,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681890840170, max timestamp=1681890840190Num uniques 1018; 2023-04-19 07:54:00,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-19 07:54:00,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681890840170, max timestamp=1681890840190Num uniques 1068; 2023-04-19 07:54:00,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-19 07:54:00,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-19 07:54:00,202 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:00,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:00,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:00,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:00,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:00,207 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:00,207 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:00,207 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:00,207 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:00,207 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,207 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,209 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:00,209 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,211 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,211 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:00,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:00,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:00,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:00,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:00,223 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:00,223 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:00,223 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:00,224 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:00,224 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:00,224 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:00,224 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:54:00,224 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:54:00,224 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:00,224 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:00,224 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:00,224 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,224 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:00,225 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:00,233 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/9d4d6e83536b4762b8b40e6cf53bfa83 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/9d4d6e83536b4762b8b40e6cf53bfa83 2023-04-19 07:54:00,236 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,236 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:00,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:00,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:00,243 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/A of 3c54cc0aa26fee1a2f14917923d1bb71 into 9d4d6e83536b4762b8b40e6cf53bfa83(size=12.7 K), total size for store is 26.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a46ccc7[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:00,243 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:00,243 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:00,243 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,243 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/A, priority=12, startTime=1681890839780; duration=0sec 2023-04-19 07:54:00,243 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:00,244 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:00,245 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:00,245 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,245 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:00,248 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:00,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:00,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:00,248 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:00,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:00,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:00,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:00,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:00,250 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:00,250 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:00,250 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:00,250 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:00,250 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:00,252 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,252 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:00,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,252 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:00,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:00,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:00,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,253 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:00,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,254 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:00,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,257 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:00,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:00,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:00,257 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:00,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:00,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:00,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,258 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:00,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:00,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:00,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:00,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,258 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:00,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,259 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:00,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:00,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890900260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681890840244, max timestamp=1681890840259Num uniques 1077; 2023-04-19 07:54:00,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-19 07:54:00,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681890840244, max timestamp=1681890840259Num uniques 1077; 2023-04-19 07:54:00,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-19 07:54:00,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681890840244, max timestamp=1681890840259Num uniques 1077; 2023-04-19 07:54:00,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-19 07:54:00,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681890840294, max timestamp=1681890840334Num uniques 1036; 2023-04-19 07:54:00,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681890840294, max timestamp=1681890840334Num uniques 1036; 2023-04-19 07:54:00,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-19 07:54:00,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681890840294, max timestamp=1681890840334Num uniques 1036; 2023-04-19 07:54:00,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-19 07:54:00,337 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-19 07:54:00,361 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:00,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:00,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:00,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:00,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:00,376 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:00,376 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:00,376 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:00,376 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:00,377 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,377 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,378 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:00,378 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:00,378 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,378 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:00,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:00,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:00,388 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:00,388 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:00,388 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:00,391 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:00,391 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:00,391 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:00,391 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:54:00,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681890840374, max timestamp=1681890840389Num uniques 1095; 2023-04-19 07:54:00,391 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:54:00,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-19 07:54:00,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681890840374, max timestamp=1681890840389Num uniques 1095; 2023-04-19 07:54:00,392 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:00,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-19 07:54:00,392 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:00,392 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:00,392 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,392 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:00,392 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:00,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681890840374, max timestamp=1681890840389Num uniques 1095; 2023-04-19 07:54:00,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-04-19 07:54:00,393 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,394 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:00,394 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,394 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:00,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:00,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ca12a26[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:00,398 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:00,398 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:00,400 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,400 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:00,401 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,401 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:00,401 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:00,401 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:00,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:00,404 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:00,404 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:00,404 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:00,404 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:00,404 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:00,406 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,406 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,406 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:00,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:00,406 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,408 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,412 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,412 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:00,412 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,412 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:00,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:00,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:00,412 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:00,412 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:00,413 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,413 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:00,413 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:00,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:00,413 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,413 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:00,413 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:00,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890900427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:00,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681890840409, max timestamp=1681890840427Num uniques 1104; 2023-04-19 07:54:00,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890900428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-19 07:54:00,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681890840409, max timestamp=1681890840427Num uniques 1104; 2023-04-19 07:54:00,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681890840409, max timestamp=1681890840427Num uniques 1104; 2023-04-19 07:54:00,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-19 07:54:00,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=11630, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-04-19 07:54:00,513 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:00,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:00,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:00,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:00,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:00,518 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:00,518 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:00,518 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:00,518 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:00,520 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,520 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,521 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:00,521 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:00,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:00,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:00,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:00,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:00,523 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:00,523 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:00,523 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:00,524 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:00,524 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:00,527 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:00,527 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:54:00,527 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:54:00,527 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:00,528 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,528 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:00,529 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,530 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:00,530 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,530 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:00,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:00,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e08b752[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:00,532 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:00,532 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,533 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/cde6202354964bbeb6fb8579017dcd3e 2023-04-19 07:54:00,534 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:00,535 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,535 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:00,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:00,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:00,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:00,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:00,542 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:00,542 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:00,542 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:00,542 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:00,542 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:00,545 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,545 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:00,545 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:00,546 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,549 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,556 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,556 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,556 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,556 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,556 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:00,556 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:00,556 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:00,556 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:00,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:00,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:00,556 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:00,556 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:00,557 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:00,557 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:00,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:00,568 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/315441ff37fb4a2d9369e1ac9fab16aa 2023-04-19 07:54:00,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681890840570, max timestamp=1681890840583Num uniques 1113; 2023-04-19 07:54:00,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-19 07:54:00,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681890840570, max timestamp=1681890840583Num uniques 1113; 2023-04-19 07:54:00,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-19 07:54:00,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681890840570, max timestamp=1681890840583Num uniques 1113; 2023-04-19 07:54:00,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-19 07:54:00,627 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=10.38 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/b7c22128a5fb40f29e60d34f2e6e8d6b 2023-04-19 07:54:00,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/cde6202354964bbeb6fb8579017dcd3e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/cde6202354964bbeb6fb8579017dcd3e 2023-04-19 07:54:00,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681890840627, max timestamp=1681890840646Num uniques 1072; 2023-04-19 07:54:00,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-19 07:54:00,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681890840627, max timestamp=1681890840646Num uniques 1022; 2023-04-19 07:54:00,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681890840627, max timestamp=1681890840646Num uniques 1072; 2023-04-19 07:54:00,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-19 07:54:00,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-19 07:54:00,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/cde6202354964bbeb6fb8579017dcd3e, entries=200, sequenceid=496, filesize=14.3 K 2023-04-19 07:54:00,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/315441ff37fb4a2d9369e1ac9fab16aa as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/315441ff37fb4a2d9369e1ac9fab16aa 2023-04-19 07:54:00,657 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:00,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:00,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:00,663 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/315441ff37fb4a2d9369e1ac9fab16aa, entries=150, sequenceid=496, filesize=11.9 K 2023-04-19 07:54:00,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:00,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:00,664 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:00,664 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:00,664 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:00,664 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:00,665 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,665 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,666 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/b7c22128a5fb40f29e60d34f2e6e8d6b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b7c22128a5fb40f29e60d34f2e6e8d6b 2023-04-19 07:54:00,668 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:00,668 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:00,669 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,669 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:00,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:00,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:00,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:00,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:00,675 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:00,675 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:00,675 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:00,678 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b7c22128a5fb40f29e60d34f2e6e8d6b, entries=150, sequenceid=496, filesize=11.9 K 2023-04-19 07:54:00,680 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~37.22 KB/38112, heapSize ~93.59 KB/95840, currentSize=80.51 KB/82440 for 3c54cc0aa26fee1a2f14917923d1bb71 in 601ms, sequenceid=496, compaction requested=true 2023-04-19 07:54:00,680 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:00,680 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:00,680 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:54:00,680 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:54:00,680 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:00,681 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:00,685 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:54:00,685 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42222 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:54:00,685 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/B is initiating minor compaction (all files) 2023-04-19 07:54:00,685 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/A is initiating minor compaction (all files) 2023-04-19 07:54:00,685 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/B in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:00,685 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/A in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:00,685 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/38f991b196e347aab26c76e9aa024496, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/aed165a0573448a1853f545edeb74e9c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/315441ff37fb4a2d9369e1ac9fab16aa] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=36.5 K 2023-04-19 07:54:00,685 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/9d4d6e83536b4762b8b40e6cf53bfa83, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/a45df8dd9d2b4b12a2bbbd08fa7258d7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/cde6202354964bbeb6fb8579017dcd3e] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=41.2 K 2023-04-19 07:54:00,686 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 38f991b196e347aab26c76e9aa024496, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681890838741 2023-04-19 07:54:00,686 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 9d4d6e83536b4762b8b40e6cf53bfa83, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681890838741 2023-04-19 07:54:00,686 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting aed165a0573448a1853f545edeb74e9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681890839882 2023-04-19 07:54:00,686 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting a45df8dd9d2b4b12a2bbbd08fa7258d7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681890839307 2023-04-19 07:54:00,687 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 315441ff37fb4a2d9369e1ac9fab16aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1681890840046 2023-04-19 07:54:00,687 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting cde6202354964bbeb6fb8579017dcd3e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1681890840046 2023-04-19 07:54:00,696 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:00,696 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:54:00,696 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=87.22 KB heapSize=244.31 KB 2023-04-19 07:54:00,701 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#B#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:00,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:54:00,701 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:54:00,712 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:54:00,712 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:00,712 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:54:00,712 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:00,712 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:54:00,712 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:00,713 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#A#compaction#79 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:00,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:00,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890900750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:00,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890900751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,769 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/62e60c92741447c19484dd05b6e3432f 2023-04-19 07:54:00,774 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/6c87604ada034854b67520f7410d9e75 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/6c87604ada034854b67520f7410d9e75 2023-04-19 07:54:00,781 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/A of 3c54cc0aa26fee1a2f14917923d1bb71 into 6c87604ada034854b67520f7410d9e75(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:00,781 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:00,781 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/A, priority=13, startTime=1681890840680; duration=0sec 2023-04-19 07:54:00,781 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:00,781 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:54:00,782 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:54:00,782 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/C is initiating minor compaction (all files) 2023-04-19 07:54:00,782 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/C in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:00,783 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/19096e4b768f4ad6aea590da6c7c0698, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/485fb7eba7bd4a79bad392b08d21c24a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b7c22128a5fb40f29e60d34f2e6e8d6b] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=36.5 K 2023-04-19 07:54:00,783 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 19096e4b768f4ad6aea590da6c7c0698, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1681890838741 2023-04-19 07:54:00,784 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 485fb7eba7bd4a79bad392b08d21c24a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681890839882 2023-04-19 07:54:00,785 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting b7c22128a5fb40f29e60d34f2e6e8d6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1681890840046 2023-04-19 07:54:00,786 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/4640738f587c46ac9f44df9dbcb8f1da 2023-04-19 07:54:00,798 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#C#compaction#82 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:00,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:00,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890900853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:00,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:00,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890900867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:01,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890901055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:01,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890901071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,142 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/678085a753cf48f89bb276bd65ce9be8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/678085a753cf48f89bb276bd65ce9be8 2023-04-19 07:54:01,149 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/B of 3c54cc0aa26fee1a2f14917923d1bb71 into 678085a753cf48f89bb276bd65ce9be8(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:01,149 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:01,149 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/B, priority=13, startTime=1681890840680; duration=0sec 2023-04-19 07:54:01,149 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:01,210 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=648 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/57f770c107dc412e937a528c3351c513 2023-04-19 07:54:01,218 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/62e60c92741447c19484dd05b6e3432f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/62e60c92741447c19484dd05b6e3432f 2023-04-19 07:54:01,220 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/78ae207c29354f039b9fdb4aa7ae6499 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/78ae207c29354f039b9fdb4aa7ae6499 2023-04-19 07:54:01,224 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/62e60c92741447c19484dd05b6e3432f, entries=150, sequenceid=648, filesize=11.9 K 2023-04-19 07:54:01,225 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/4640738f587c46ac9f44df9dbcb8f1da as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4640738f587c46ac9f44df9dbcb8f1da 2023-04-19 07:54:01,227 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/C of 3c54cc0aa26fee1a2f14917923d1bb71 into 78ae207c29354f039b9fdb4aa7ae6499(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:01,227 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:01,227 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/C, priority=13, startTime=1681890840681; duration=0sec 2023-04-19 07:54:01,227 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:01,232 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4640738f587c46ac9f44df9dbcb8f1da, entries=150, sequenceid=648, filesize=11.9 K 2023-04-19 07:54:01,233 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/57f770c107dc412e937a528c3351c513 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/57f770c107dc412e937a528c3351c513 2023-04-19 07:54:01,238 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/57f770c107dc412e937a528c3351c513, entries=150, sequenceid=648, filesize=11.9 K 2023-04-19 07:54:01,239 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~264.19 KB/270528, currentSize=87.22 KB/89310 for 3c54cc0aa26fee1a2f14917923d1bb71 in 543ms, sequenceid=648, compaction requested=false 2023-04-19 07:54:01,239 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:01,239 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:01,239 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:01,241 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,241 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,241 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,241 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:01,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d5b58ff[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:01,243 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:01,243 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,244 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:01,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:01,244 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,244 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:01,246 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,246 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:01,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:01,249 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:01,249 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:01,249 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:01,249 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:01,249 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:01,251 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,251 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:01,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,252 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,252 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,254 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:01,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:01,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:01,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,258 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:01,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:01,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:01,259 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:01,259 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:01,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:01,359 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:01,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:01,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:01,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:01,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:01,363 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:01,363 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:01,363 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:01,363 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:01,364 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,364 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:54:01,365 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-19 07:54:01,365 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:54:01,365 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:01,365 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:54:01,365 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:01,365 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:54:01,365 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:01,366 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:01,366 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:01,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:01,366 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,366 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:01,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:01,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:01,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:01,371 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:01,371 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:01,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/fed6e3151db34f48b9c452a5ea87655c 2023-04-19 07:54:01,371 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:01,380 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:54:01,380 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:54:01,380 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:01,380 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:01,380 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:01,380 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,380 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:01,380 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,383 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:01,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11879167[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:01,388 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:01,388 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,389 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:01,390 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,390 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:01,391 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,392 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,392 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:01,392 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:01,395 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:01,395 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:01,395 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:01,395 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:01,395 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:01,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:01,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,400 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,402 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,408 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:01,408 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:01,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:01,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,409 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:01,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:01,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:01,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:01,409 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:01,409 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:01,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:01,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890901409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890901409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,510 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:01,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:01,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:01,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:01,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890901513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:01,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:01,514 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:01,514 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:01,514 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:01,514 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:01,515 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,515 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:01,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890901514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,516 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,516 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:01,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:01,516 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,516 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:01,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:01,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:01,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:01,518 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:01,518 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:01,518 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:01,518 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:01,518 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:01,518 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:01,519 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:54:01,519 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:54:01,519 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:01,519 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:01,519 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:01,519 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,519 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:01,519 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:01,527 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,527 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,527 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,527 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:01,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:01,529 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:01,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54b2796b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:01,529 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,530 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,530 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:01,530 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:01,530 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,530 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:01,530 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:01,531 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,531 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:01,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,531 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:01,532 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:01,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:01,534 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:01,534 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:01,534 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:01,534 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:01,534 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:01,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:01,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,536 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,538 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,543 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,543 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,543 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:01,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:01,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:01,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:01,543 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:01,544 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:01,543 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,544 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,544 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,544 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,544 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:01,544 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:01,544 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:01,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:01,645 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:01,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:01,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:01,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:01,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:01,649 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:01,649 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:01,650 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:01,650 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:01,650 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,650 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,651 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,652 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,652 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:01,662 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:01,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:01,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:01,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:01,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:01,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:01,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:01,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:01,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:01,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:01,664 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:01,664 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:54:01,664 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:54:01,664 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:01,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:01,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:01,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:01,665 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:01,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,666 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,667 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,667 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:01,668 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:01,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f9bf688[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:01,668 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,671 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,671 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:01,671 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:01,671 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:01,671 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,671 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:01,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,673 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:01,673 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:01,673 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:01,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:01,679 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:01,679 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:01,679 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:01,679 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:01,679 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:01,681 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,681 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,681 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,684 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,686 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,688 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,688 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,688 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:01,688 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:01,688 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:01,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:01,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:01,688 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:01,688 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,689 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:01,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,689 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:01,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:01,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:01,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890901716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,718 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:01,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890901716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,801 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:01,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:01,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:01,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:01,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:01,805 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:01,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:01,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:01,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:01,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,806 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,813 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:01,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:01,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:01,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:01,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:01,814 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:01,814 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:01,814 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:01,815 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:01,815 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:54:01,815 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:54:01,815 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:01,815 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:01,817 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,817 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,817 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,817 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7830c324[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:01,819 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:01,819 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,820 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:01,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:01,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:01,820 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,820 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:01,821 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,821 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:01,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,821 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:01,822 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,823 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/588fe2ba7fbc497fbea28d7815dc019a 2023-04-19 07:54:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:01,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:01,825 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:01,825 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:01,825 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:01,825 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:01,825 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:01,828 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,828 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:01,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:01,829 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,830 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:01,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,838 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:01,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,842 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,842 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:01,842 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:01,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:01,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:01,842 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,842 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,842 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,843 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:01,843 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,842 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:01,842 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:01,843 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:01,843 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:01,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:01,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:01,852 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/260ca567b16246dbbd29383b73545983 2023-04-19 07:54:01,858 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/fed6e3151db34f48b9c452a5ea87655c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/fed6e3151db34f48b9c452a5ea87655c 2023-04-19 07:54:01,864 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/fed6e3151db34f48b9c452a5ea87655c, entries=200, sequenceid=668, filesize=14.3 K 2023-04-19 07:54:01,868 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/588fe2ba7fbc497fbea28d7815dc019a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/588fe2ba7fbc497fbea28d7815dc019a 2023-04-19 07:54:01,874 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/588fe2ba7fbc497fbea28d7815dc019a, entries=150, sequenceid=668, filesize=11.9 K 2023-04-19 07:54:01,876 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/260ca567b16246dbbd29383b73545983 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/260ca567b16246dbbd29383b73545983 2023-04-19 07:54:01,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/260ca567b16246dbbd29383b73545983, entries=150, sequenceid=668, filesize=11.9 K 2023-04-19 07:54:01,883 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 3c54cc0aa26fee1a2f14917923d1bb71 in 518ms, sequenceid=668, compaction requested=true 2023-04-19 07:54:01,883 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:01,883 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:01,883 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:54:01,883 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:01,883 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:54:01,884 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:01,886 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:54:01,886 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:54:01,886 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/A is initiating minor compaction (all files) 2023-04-19 07:54:01,886 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/B is initiating minor compaction (all files) 2023-04-19 07:54:01,886 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/A in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:01,886 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/B in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:01,886 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/678085a753cf48f89bb276bd65ce9be8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4640738f587c46ac9f44df9dbcb8f1da, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/588fe2ba7fbc497fbea28d7815dc019a] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=36.6 K 2023-04-19 07:54:01,886 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/6c87604ada034854b67520f7410d9e75, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/62e60c92741447c19484dd05b6e3432f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/fed6e3151db34f48b9c452a5ea87655c] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=38.9 K 2023-04-19 07:54:01,887 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 678085a753cf48f89bb276bd65ce9be8, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1681890840046 2023-04-19 07:54:01,887 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 6c87604ada034854b67520f7410d9e75, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1681890840046 2023-04-19 07:54:01,887 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 4640738f587c46ac9f44df9dbcb8f1da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681890840671 2023-04-19 07:54:01,888 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 588fe2ba7fbc497fbea28d7815dc019a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=668, earliestPutTs=1681890840748 2023-04-19 07:54:01,888 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 62e60c92741447c19484dd05b6e3432f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681890840671 2023-04-19 07:54:01,888 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting fed6e3151db34f48b9c452a5ea87655c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=668, earliestPutTs=1681890840745 2023-04-19 07:54:01,899 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#B#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:01,901 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#A#compaction#88 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:01,943 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:01,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:01,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:01,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:01,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:01,946 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:01,947 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:01,947 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:01,947 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:01,947 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,947 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,948 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:01,948 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:01,949 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:01,949 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:01,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:01,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:01,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:01,950 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:01,950 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:01,950 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:01,950 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:01,950 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:01,950 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:01,950 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:54:01,951 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 07:54:01,951 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:54:01,951 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:01,951 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:54:01,951 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:01,951 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:54:01,951 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:01,960 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/62465e0db055451f90f74f5aacf09261 2023-04-19 07:54:02,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:54:02,022 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:54:02,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:02,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890902042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:02,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890902044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:02,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890902147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:02,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890902150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,320 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/6c948afba99f4d3ca6ea967d6409dd6e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6c948afba99f4d3ca6ea967d6409dd6e 2023-04-19 07:54:02,328 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/e8f86f2d016d4524a7dd896277e9c24e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e8f86f2d016d4524a7dd896277e9c24e 2023-04-19 07:54:02,328 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/B of 3c54cc0aa26fee1a2f14917923d1bb71 into 6c948afba99f4d3ca6ea967d6409dd6e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:02,328 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:02,328 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/B, priority=13, startTime=1681890841883; duration=0sec 2023-04-19 07:54:02,328 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:02,328 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:54:02,331 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:54:02,331 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/C is initiating minor compaction (all files) 2023-04-19 07:54:02,331 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/C in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:02,332 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/78ae207c29354f039b9fdb4aa7ae6499, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/57f770c107dc412e937a528c3351c513, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/260ca567b16246dbbd29383b73545983] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=36.6 K 2023-04-19 07:54:02,332 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 78ae207c29354f039b9fdb4aa7ae6499, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1681890840046 2023-04-19 07:54:02,333 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 57f770c107dc412e937a528c3351c513, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=648, earliestPutTs=1681890840671 2023-04-19 07:54:02,334 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 260ca567b16246dbbd29383b73545983, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=668, earliestPutTs=1681890840748 2023-04-19 07:54:02,336 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/A of 3c54cc0aa26fee1a2f14917923d1bb71 into e8f86f2d016d4524a7dd896277e9c24e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:02,336 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:02,336 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/A, priority=13, startTime=1681890841883; duration=0sec 2023-04-19 07:54:02,336 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:02,344 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#C#compaction#91 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:02,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:02,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:02,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890902351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890902352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,364 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/673cc4dd3e02417fabdfd01499371639 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/673cc4dd3e02417fabdfd01499371639 2023-04-19 07:54:02,370 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/C of 3c54cc0aa26fee1a2f14917923d1bb71 into 673cc4dd3e02417fabdfd01499371639(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:02,370 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:02,370 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/C, priority=13, startTime=1681890841884; duration=0sec 2023-04-19 07:54:02,371 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:02,374 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/a2ebc5faf069435da2b015f5e76e563c 2023-04-19 07:54:02,393 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=684 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/64156c60056043c3b367a0fe7e55a2d0 2023-04-19 07:54:02,399 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/62465e0db055451f90f74f5aacf09261 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/62465e0db055451f90f74f5aacf09261 2023-04-19 07:54:02,404 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/62465e0db055451f90f74f5aacf09261, entries=150, sequenceid=684, filesize=11.9 K 2023-04-19 07:54:02,405 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/a2ebc5faf069435da2b015f5e76e563c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/a2ebc5faf069435da2b015f5e76e563c 2023-04-19 07:54:02,412 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/a2ebc5faf069435da2b015f5e76e563c, entries=150, sequenceid=684, filesize=11.9 K 2023-04-19 07:54:02,413 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/64156c60056043c3b367a0fe7e55a2d0 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/64156c60056043c3b367a0fe7e55a2d0 2023-04-19 07:54:02,418 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/64156c60056043c3b367a0fe7e55a2d0, entries=150, sequenceid=684, filesize=11.9 K 2023-04-19 07:54:02,420 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 3c54cc0aa26fee1a2f14917923d1bb71 in 470ms, sequenceid=684, compaction requested=false 2023-04-19 07:54:02,421 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:02,421 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:02,421 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:02,421 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:02,421 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,421 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:02,421 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:02,433 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,433 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:02,433 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:02,433 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:02,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:02,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:02,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@551169ec[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:02,436 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:02,436 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:02,437 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:02,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:02,437 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,437 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:02,439 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:02,439 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:02,439 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:02,439 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:02,444 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:02,444 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:02,444 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:02,444 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:02,444 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:02,446 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:02,446 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:02,446 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:02,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:02,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:02,447 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,448 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,451 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,451 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:02,451 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:02,451 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:02,451 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:02,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:02,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:02,451 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:02,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:02,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:02,451 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,451 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:02,451 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:02,451 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:02,451 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:02,451 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:02,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:02,551 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:02,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:02,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:02,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:02,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:02,558 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:02,558 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:02,558 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:02,558 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:02,558 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:02,558 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:02,571 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:02,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:02,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:02,572 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,572 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,572 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:02,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:02,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:02,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:02,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:02,574 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:02,574 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:02,574 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:02,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:02,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:02,575 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:02,575 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:54:02,575 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 07:54:02,575 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:54:02,576 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:02,576 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:54:02,576 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:02,576 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:54:02,576 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:02,598 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/911f8f07b61f4aa29ce47d31db2164b2 2023-04-19 07:54:02,628 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/0e9dfa82356942e28bd797f5b9b8a9d7 2023-04-19 07:54:02,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:54:02,656 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:54:02,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:02,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890902677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:02,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890902679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:02,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890902784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:02,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890902785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:02,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890902987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:02,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:02,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890902990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:03,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57322 deadline: 1681890903037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,040 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:03,054 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=703 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/ac9eea73c7b04feeaa39f62fa13788d4 2023-04-19 07:54:03,062 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/911f8f07b61f4aa29ce47d31db2164b2 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/911f8f07b61f4aa29ce47d31db2164b2 2023-04-19 07:54:03,067 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/911f8f07b61f4aa29ce47d31db2164b2, entries=150, sequenceid=703, filesize=11.9 K 2023-04-19 07:54:03,069 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/0e9dfa82356942e28bd797f5b9b8a9d7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/0e9dfa82356942e28bd797f5b9b8a9d7 2023-04-19 07:54:03,076 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/0e9dfa82356942e28bd797f5b9b8a9d7, entries=150, sequenceid=703, filesize=11.9 K 2023-04-19 07:54:03,077 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/ac9eea73c7b04feeaa39f62fa13788d4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/ac9eea73c7b04feeaa39f62fa13788d4 2023-04-19 07:54:03,089 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/ac9eea73c7b04feeaa39f62fa13788d4, entries=150, sequenceid=703, filesize=11.9 K 2023-04-19 07:54:03,090 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 3c54cc0aa26fee1a2f14917923d1bb71 in 515ms, sequenceid=703, compaction requested=true 2023-04-19 07:54:03,090 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:03,090 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:03,090 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:03,090 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:03,090 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,091 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:03,091 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:03,092 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:03,092 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:03,093 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:03,093 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:03,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:03,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@122f3b04[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:03,096 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:03,096 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:03,097 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:03,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:03,097 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:03,097 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:03,097 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:03,097 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,097 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:03,099 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:03,099 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:03,099 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,099 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:03,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:03,102 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:03,102 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:03,102 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:03,102 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:03,102 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:03,105 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:03,105 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:03,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:03,105 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:03,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:03,105 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,107 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:03,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:03,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:03,110 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:03,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:03,110 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,110 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:03,110 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:03,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:03,110 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:03,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:03,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:03,110 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:03,110 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:03,110 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:03,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:03,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:03,210 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:03,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:03,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:03,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:03,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:03,215 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:03,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:03,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:03,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:03,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:03,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:03,216 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:03,217 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:03,217 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,217 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:03,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:03,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:03,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:03,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:03,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:03,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:03,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:03,220 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:03,220 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:03,220 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:03,220 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:54:03,220 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 07:54:03,220 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:54:03,221 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:03,221 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:54:03,221 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:03,221 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:54:03,221 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:03,236 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/bd1b18ee7f3740dd8977a0cd3f5e21e8 2023-04-19 07:54:03,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:54:03,294 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:54:03,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:03,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890903312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:03,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890903313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:03,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890903415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:03,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890903422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:03,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890903620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:03,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890903624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,648 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/0d1975c9c1a74c4aa9d31cf40c17e0a6 2023-04-19 07:54:03,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:03,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890903924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:03,931 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:03,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890903929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,061 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/0585128365db49d9a6356179ca1dfea9 2023-04-19 07:54:04,072 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/bd1b18ee7f3740dd8977a0cd3f5e21e8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/bd1b18ee7f3740dd8977a0cd3f5e21e8 2023-04-19 07:54:04,078 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/bd1b18ee7f3740dd8977a0cd3f5e21e8, entries=150, sequenceid=719, filesize=11.9 K 2023-04-19 07:54:04,080 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/0d1975c9c1a74c4aa9d31cf40c17e0a6 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/0d1975c9c1a74c4aa9d31cf40c17e0a6 2023-04-19 07:54:04,086 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/0d1975c9c1a74c4aa9d31cf40c17e0a6, entries=150, sequenceid=719, filesize=11.9 K 2023-04-19 07:54:04,087 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/0585128365db49d9a6356179ca1dfea9 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0585128365db49d9a6356179ca1dfea9 2023-04-19 07:54:04,101 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0585128365db49d9a6356179ca1dfea9, entries=150, sequenceid=719, filesize=11.9 K 2023-04-19 07:54:04,102 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 3c54cc0aa26fee1a2f14917923d1bb71 in 882ms, sequenceid=719, compaction requested=true 2023-04-19 07:54:04,102 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:04,102 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:04,102 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:04,102 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:04,102 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,104 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:04,104 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:04,105 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,105 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:04,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:04,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:04,106 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:04,106 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:04,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:04,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:04,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:04,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:04,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:04,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@290a07f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:04,108 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:04,108 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:04,110 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:04,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:04,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:04,111 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:04,111 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:04,111 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:04,112 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,113 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:04,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:04,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:04,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:04,114 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:04,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:04,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:04,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:04,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:04,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:04,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:04,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:04,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:04,118 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:04,118 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:04,118 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:04,118 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:04,118 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:04,120 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:04,120 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:04,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:04,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:04,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:04,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:04,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:04,120 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:04,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:04,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:04,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:04,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:04,121 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,123 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,125 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,125 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:04,125 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:04,125 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:04,125 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,125 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:04,125 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:04,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:04,125 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:04,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:04,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:04,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:04,125 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:04,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:04,126 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:04,126 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:04,126 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:04,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:04,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:04,226 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:04,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:04,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:04,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:04,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:04,239 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:04,239 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:04,239 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:04,239 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:04,240 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:04,240 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:04,242 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:04,242 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:04,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:04,242 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,243 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:04,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:04,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:04,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:04,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:04,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:04,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:04,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:04,256 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:04,256 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:04,256 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:04,256 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. started... 2023-04-19 07:54:04,256 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 07:54:04,256 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:54:04,257 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:04,257 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:54:04,257 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:04,257 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:54:04,257 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:04,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:54:04,429 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. as already flushing 2023-04-19 07:54:04,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:04,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890904451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:04,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890904452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,498 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2879e2e1 to 127.0.0.1:58087 2023-04-19 07:54:04,498 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:04,499 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49d71120 to 127.0.0.1:58087 2023-04-19 07:54:04,499 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:04,500 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x60b8c4c7 to 127.0.0.1:58087 2023-04-19 07:54:04,500 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:04,504 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d24bbb7 to 127.0.0.1:58087 2023-04-19 07:54:04,504 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:04,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:04,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890904554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:04,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890904555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,674 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/30f9b4b6650d44738cc829c6c80e6d0b 2023-04-19 07:54:04,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:04,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 420 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890904757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:04,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:04,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890904757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:05,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:05,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681890905060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57262 deadline: 1681890905060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,093 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/cf46a7d2edb54d3faa591281a2b4d61a 2023-04-19 07:54:05,505 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/34f3683733e24f9c9d256f0de26ce466 2023-04-19 07:54:05,511 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/30f9b4b6650d44738cc829c6c80e6d0b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/30f9b4b6650d44738cc829c6c80e6d0b 2023-04-19 07:54:05,516 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/30f9b4b6650d44738cc829c6c80e6d0b, entries=100, sequenceid=735, filesize=9.5 K 2023-04-19 07:54:05,517 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/cf46a7d2edb54d3faa591281a2b4d61a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/cf46a7d2edb54d3faa591281a2b4d61a 2023-04-19 07:54:05,522 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/cf46a7d2edb54d3faa591281a2b4d61a, entries=100, sequenceid=735, filesize=9.5 K 2023-04-19 07:54:05,523 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/34f3683733e24f9c9d256f0de26ce466 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/34f3683733e24f9c9d256f0de26ce466 2023-04-19 07:54:05,528 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/34f3683733e24f9c9d256f0de26ce466, entries=100, sequenceid=735, filesize=9.5 K 2023-04-19 07:54:05,529 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 3c54cc0aa26fee1a2f14917923d1bb71 in 1273ms, sequenceid=735, compaction requested=true 2023-04-19 07:54:05,529 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:05,529 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:05,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:05,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:05,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:05,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:05,531 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:05,531 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:05,531 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:05,531 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:05,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:05,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:05,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c477aea[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:05,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:05,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:05,535 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:05,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:05,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:05,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:05,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:05,535 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,535 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:05,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:05,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:05,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:05,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:05,539 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:05,539 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:05,539 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:05,539 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:05,539 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:05,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:05,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:05,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:05,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:05,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:05,541 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:05,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:05,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:05,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:05,543 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:05,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,545 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:05,545 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:05,545 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,545 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:05,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:05,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:05,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:05,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:05,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:05,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:05,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:05,546 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:05,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:05,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:05,546 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:05,546 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:05,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:05,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:54:05,567 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-19 07:54:05,568 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x748f869b to 127.0.0.1:58087 2023-04-19 07:54:05,568 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46192d17 to 127.0.0.1:58087 2023-04-19 07:54:05,568 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:05,568 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:54:05,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:05,568 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:54:05,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:05,568 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:54:05,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:05,568 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:05,594 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e91d243 to 127.0.0.1:58087 2023-04-19 07:54:05,594 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:05,643 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79eb61dd to 127.0.0.1:58087 2023-04-19 07:54:05,643 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:05,646 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:05,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:05,976 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=753 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/3c2e72e0c6424c129cc38f0a6636298a 2023-04-19 07:54:06,387 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=753 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/6873e1c6674f4c268ad77f41fb4d3b6a 2023-04-19 07:54:06,798 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=753 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/908dd5b4050b410aab11eee79354bc43 2023-04-19 07:54:06,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/3c2e72e0c6424c129cc38f0a6636298a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/3c2e72e0c6424c129cc38f0a6636298a 2023-04-19 07:54:06,807 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/3c2e72e0c6424c129cc38f0a6636298a, entries=150, sequenceid=753, filesize=11.9 K 2023-04-19 07:54:06,808 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/6873e1c6674f4c268ad77f41fb4d3b6a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6873e1c6674f4c268ad77f41fb4d3b6a 2023-04-19 07:54:06,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6873e1c6674f4c268ad77f41fb4d3b6a, entries=150, sequenceid=753, filesize=11.9 K 2023-04-19 07:54:06,812 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/908dd5b4050b410aab11eee79354bc43 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/908dd5b4050b410aab11eee79354bc43 2023-04-19 07:54:06,817 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/908dd5b4050b410aab11eee79354bc43, entries=150, sequenceid=753, filesize=11.9 K 2023-04-19 07:54:06,817 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=13.42 KB/13740 for 3c54cc0aa26fee1a2f14917923d1bb71 in 1250ms, sequenceid=753, compaction requested=true 2023-04-19 07:54:06,818 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:06,818 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:06,818 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:06,818 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 07:54:06,818 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 07:54:06,818 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:06,820 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71660 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 07:54:06,820 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/A is initiating minor compaction (all files) 2023-04-19 07:54:06,820 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71660 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 07:54:06,820 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/A in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:06,820 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/B is initiating minor compaction (all files) 2023-04-19 07:54:06,820 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e8f86f2d016d4524a7dd896277e9c24e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/62465e0db055451f90f74f5aacf09261, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/911f8f07b61f4aa29ce47d31db2164b2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/bd1b18ee7f3740dd8977a0cd3f5e21e8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/30f9b4b6650d44738cc829c6c80e6d0b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/3c2e72e0c6424c129cc38f0a6636298a] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=70.0 K 2023-04-19 07:54:06,820 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/B in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:06,820 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6c948afba99f4d3ca6ea967d6409dd6e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/a2ebc5faf069435da2b015f5e76e563c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/0e9dfa82356942e28bd797f5b9b8a9d7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/0d1975c9c1a74c4aa9d31cf40c17e0a6, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/cf46a7d2edb54d3faa591281a2b4d61a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6873e1c6674f4c268ad77f41fb4d3b6a] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=70.0 K 2023-04-19 07:54:06,820 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting e8f86f2d016d4524a7dd896277e9c24e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=668, earliestPutTs=1681890840748 2023-04-19 07:54:06,820 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 6c948afba99f4d3ca6ea967d6409dd6e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=668, earliestPutTs=1681890840748 2023-04-19 07:54:06,821 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 62465e0db055451f90f74f5aacf09261, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1681890841393 2023-04-19 07:54:06,821 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting a2ebc5faf069435da2b015f5e76e563c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1681890841393 2023-04-19 07:54:06,821 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 911f8f07b61f4aa29ce47d31db2164b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1681890842035 2023-04-19 07:54:06,821 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 0e9dfa82356942e28bd797f5b9b8a9d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1681890842035 2023-04-19 07:54:06,821 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting bd1b18ee7f3740dd8977a0cd3f5e21e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1681890842663 2023-04-19 07:54:06,821 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 0d1975c9c1a74c4aa9d31cf40c17e0a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1681890842663 2023-04-19 07:54:06,822 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 30f9b4b6650d44738cc829c6c80e6d0b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1681890843309 2023-04-19 07:54:06,822 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting cf46a7d2edb54d3faa591281a2b4d61a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1681890843309 2023-04-19 07:54:06,822 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 3c2e72e0c6424c129cc38f0a6636298a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1681890844447 2023-04-19 07:54:06,822 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 6873e1c6674f4c268ad77f41fb4d3b6a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1681890844447 2023-04-19 07:54:06,835 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#B#compaction#106 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:06,835 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#A#compaction#105 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:07,248 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/42b382c056fd4a8ab492bd98ccec1822 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/42b382c056fd4a8ab492bd98ccec1822 2023-04-19 07:54:07,252 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/e3f932241ef441c0bbf7988d35a65df5 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/e3f932241ef441c0bbf7988d35a65df5 2023-04-19 07:54:07,254 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/A of 3c54cc0aa26fee1a2f14917923d1bb71 into 42b382c056fd4a8ab492bd98ccec1822(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:07,254 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:07,254 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/A, priority=10, startTime=1681890846818; duration=0sec 2023-04-19 07:54:07,254 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:07,254 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 07:54:07,256 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71660 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 07:54:07,256 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 3c54cc0aa26fee1a2f14917923d1bb71/C is initiating minor compaction (all files) 2023-04-19 07:54:07,256 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3c54cc0aa26fee1a2f14917923d1bb71/C in TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:07,256 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/673cc4dd3e02417fabdfd01499371639, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/64156c60056043c3b367a0fe7e55a2d0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/ac9eea73c7b04feeaa39f62fa13788d4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0585128365db49d9a6356179ca1dfea9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/34f3683733e24f9c9d256f0de26ce466, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/908dd5b4050b410aab11eee79354bc43] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp, totalSize=70.0 K 2023-04-19 07:54:07,257 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 673cc4dd3e02417fabdfd01499371639, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=668, earliestPutTs=1681890840748 2023-04-19 07:54:07,257 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 64156c60056043c3b367a0fe7e55a2d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=684, earliestPutTs=1681890841393 2023-04-19 07:54:07,258 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting ac9eea73c7b04feeaa39f62fa13788d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=703, earliestPutTs=1681890842035 2023-04-19 07:54:07,258 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/B of 3c54cc0aa26fee1a2f14917923d1bb71 into e3f932241ef441c0bbf7988d35a65df5(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:07,258 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:07,258 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/B, priority=10, startTime=1681890846818; duration=0sec 2023-04-19 07:54:07,258 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 0585128365db49d9a6356179ca1dfea9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1681890842663 2023-04-19 07:54:07,258 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:07,259 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 34f3683733e24f9c9d256f0de26ce466, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=735, earliestPutTs=1681890843309 2023-04-19 07:54:07,259 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 908dd5b4050b410aab11eee79354bc43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1681890844447 2023-04-19 07:54:07,269 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3c54cc0aa26fee1a2f14917923d1bb71#C#compaction#107 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:07,682 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/12ace067c01d4a7e8405d854c019c76b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/12ace067c01d4a7e8405d854c019c76b 2023-04-19 07:54:07,688 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3c54cc0aa26fee1a2f14917923d1bb71/C of 3c54cc0aa26fee1a2f14917923d1bb71 into 12ace067c01d4a7e8405d854c019c76b(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:07,688 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:07,689 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71., storeName=3c54cc0aa26fee1a2f14917923d1bb71/C, priority=10, startTime=1681890846818; duration=0sec 2023-04-19 07:54:07,689 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:13,097 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30eb7fb2 to 127.0.0.1:58087 2023-04-19 07:54:13,097 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 270 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 21 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 81 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 256 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(398): read 3615 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(398): read 3647 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(402): scanned 1611 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(403): verified 4833 rows 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(402): scanned 1617 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(403): verified 4845 rows 2023-04-19 07:54:13,098 INFO [Listener at localhost/42517] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-19 07:54:13,099 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d9fc901 to 127.0.0.1:58087 2023-04-19 07:54:13,099 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:13,104 INFO [Listener at localhost/42517] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-19 07:54:13,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-19 07:54:13,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:13,124 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890853124"}]},"ts":"1681890853124"} 2023-04-19 07:54:13,126 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-19 07:54:13,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-19 07:54:13,204 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-19 07:54:13,207 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3c54cc0aa26fee1a2f14917923d1bb71, UNASSIGN}] 2023-04-19 07:54:13,209 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3c54cc0aa26fee1a2f14917923d1bb71, UNASSIGN 2023-04-19 07:54:13,210 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=3c54cc0aa26fee1a2f14917923d1bb71, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:13,211 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890853210"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890853210"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890853210"}]},"ts":"1681890853210"} 2023-04-19 07:54:13,213 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:54:13,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-19 07:54:13,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:54:13,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3c54cc0aa26fee1a2f14917923d1bb71, disabling compactions & flushes 2023-04-19 07:54:13,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:13,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:13,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. after waiting 0 ms 2023-04-19 07:54:13,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:13,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 3c54cc0aa26fee1a2f14917923d1bb71 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-19 07:54:13,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=A 2023-04-19 07:54:13,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:13,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=B 2023-04-19 07:54:13,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:13,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3c54cc0aa26fee1a2f14917923d1bb71, store=C 2023-04-19 07:54:13,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:13,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-19 07:54:13,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-19 07:54:13,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=762 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/709ef7ae0e744eaebc22bf4243c13cff 2023-04-19 07:54:14,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=762 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/2cc9ece63af94b20a976ff69e6bd163e 2023-04-19 07:54:14,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-19 07:54:14,607 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=762 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/08fe0bebe1f64e7c9ac29d8c49be4bef 2023-04-19 07:54:14,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/A/709ef7ae0e744eaebc22bf4243c13cff as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/709ef7ae0e744eaebc22bf4243c13cff 2023-04-19 07:54:14,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/709ef7ae0e744eaebc22bf4243c13cff, entries=100, sequenceid=762, filesize=9.5 K 2023-04-19 07:54:14,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/B/2cc9ece63af94b20a976ff69e6bd163e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/2cc9ece63af94b20a976ff69e6bd163e 2023-04-19 07:54:14,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/2cc9ece63af94b20a976ff69e6bd163e, entries=100, sequenceid=762, filesize=9.5 K 2023-04-19 07:54:14,621 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/.tmp/C/08fe0bebe1f64e7c9ac29d8c49be4bef as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/08fe0bebe1f64e7c9ac29d8c49be4bef 2023-04-19 07:54:14,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/08fe0bebe1f64e7c9ac29d8c49be4bef, entries=100, sequenceid=762, filesize=9.5 K 2023-04-19 07:54:14,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 3c54cc0aa26fee1a2f14917923d1bb71 in 1252ms, sequenceid=762, compaction requested=false 2023-04-19 07:54:14,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e4f359cd9a2e4ccba791ba7c9d2be373, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/0626f31d96fd4c33a2e9fc64b1b65201, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/fb0daf70a1d74aeb997822cdc7c59dc2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5af29cb5a72349c888437dc69e9bd592, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b2220ff6c478474f89a4641a22f44fb7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/1cdd7fce798a4f73a47bf7d88060f5dd, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/526fae089dc648a9aa15c579d2de0881, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/6daf8d58af844023b3745815f67e872c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/7709fcfcad6c4be997206ed7dcda77ff, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e36897b2d41548c5839fb4405a9a53c6, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/629e655354664747b9ed4f04535e40a7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/7d2a9c4a00d040f9983c72d42add06ec, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5e0ab40f3b86448b9c1cb49dc1a6d6fb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/a777045aab7b4c758ce82b93c21a15ce, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b6701425e8aa40ba9225d91dae946824, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b095d5e158ca4202b2bcd164605d710e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/27c42c3122ee45edaebdd9a21676c3f2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/d16492b35c5f4ae4b5fde196ef781775, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/ab05e2b84e724a51989e7f9f9e168b9a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/8c55ea119cfb49f789effc59c8d4bcb9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/dc42a823b32b4397ba01bd0d1d3effe9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/3cda03775dc0482598c8500181356856, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/9d4d6e83536b4762b8b40e6cf53bfa83, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5b42b0b64928408ba7bd001bd709f293, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/a45df8dd9d2b4b12a2bbbd08fa7258d7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/cde6202354964bbeb6fb8579017dcd3e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/6c87604ada034854b67520f7410d9e75, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/62e60c92741447c19484dd05b6e3432f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/fed6e3151db34f48b9c452a5ea87655c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e8f86f2d016d4524a7dd896277e9c24e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/62465e0db055451f90f74f5aacf09261, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/911f8f07b61f4aa29ce47d31db2164b2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/bd1b18ee7f3740dd8977a0cd3f5e21e8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/30f9b4b6650d44738cc829c6c80e6d0b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/3c2e72e0c6424c129cc38f0a6636298a] to archive 2023-04-19 07:54:14,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:54:14,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e4f359cd9a2e4ccba791ba7c9d2be373 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e4f359cd9a2e4ccba791ba7c9d2be373 2023-04-19 07:54:14,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/0626f31d96fd4c33a2e9fc64b1b65201 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/0626f31d96fd4c33a2e9fc64b1b65201 2023-04-19 07:54:14,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/fb0daf70a1d74aeb997822cdc7c59dc2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/fb0daf70a1d74aeb997822cdc7c59dc2 2023-04-19 07:54:14,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5af29cb5a72349c888437dc69e9bd592 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5af29cb5a72349c888437dc69e9bd592 2023-04-19 07:54:14,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b2220ff6c478474f89a4641a22f44fb7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b2220ff6c478474f89a4641a22f44fb7 2023-04-19 07:54:14,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/1cdd7fce798a4f73a47bf7d88060f5dd to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/1cdd7fce798a4f73a47bf7d88060f5dd 2023-04-19 07:54:14,660 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/526fae089dc648a9aa15c579d2de0881 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/526fae089dc648a9aa15c579d2de0881 2023-04-19 07:54:14,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/6daf8d58af844023b3745815f67e872c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/6daf8d58af844023b3745815f67e872c 2023-04-19 07:54:14,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/7709fcfcad6c4be997206ed7dcda77ff to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/7709fcfcad6c4be997206ed7dcda77ff 2023-04-19 07:54:14,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e36897b2d41548c5839fb4405a9a53c6 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e36897b2d41548c5839fb4405a9a53c6 2023-04-19 07:54:14,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/629e655354664747b9ed4f04535e40a7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/629e655354664747b9ed4f04535e40a7 2023-04-19 07:54:14,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/7d2a9c4a00d040f9983c72d42add06ec to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/7d2a9c4a00d040f9983c72d42add06ec 2023-04-19 07:54:14,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5e0ab40f3b86448b9c1cb49dc1a6d6fb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5e0ab40f3b86448b9c1cb49dc1a6d6fb 2023-04-19 07:54:14,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/a777045aab7b4c758ce82b93c21a15ce to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/a777045aab7b4c758ce82b93c21a15ce 2023-04-19 07:54:14,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b6701425e8aa40ba9225d91dae946824 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b6701425e8aa40ba9225d91dae946824 2023-04-19 07:54:14,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b095d5e158ca4202b2bcd164605d710e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/b095d5e158ca4202b2bcd164605d710e 2023-04-19 07:54:14,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/27c42c3122ee45edaebdd9a21676c3f2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/27c42c3122ee45edaebdd9a21676c3f2 2023-04-19 07:54:14,674 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/d16492b35c5f4ae4b5fde196ef781775 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/d16492b35c5f4ae4b5fde196ef781775 2023-04-19 07:54:14,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/ab05e2b84e724a51989e7f9f9e168b9a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/ab05e2b84e724a51989e7f9f9e168b9a 2023-04-19 07:54:14,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/8c55ea119cfb49f789effc59c8d4bcb9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/8c55ea119cfb49f789effc59c8d4bcb9 2023-04-19 07:54:14,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/dc42a823b32b4397ba01bd0d1d3effe9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/dc42a823b32b4397ba01bd0d1d3effe9 2023-04-19 07:54:14,679 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/3cda03775dc0482598c8500181356856 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/3cda03775dc0482598c8500181356856 2023-04-19 07:54:14,680 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/9d4d6e83536b4762b8b40e6cf53bfa83 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/9d4d6e83536b4762b8b40e6cf53bfa83 2023-04-19 07:54:14,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5b42b0b64928408ba7bd001bd709f293 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/5b42b0b64928408ba7bd001bd709f293 2023-04-19 07:54:14,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/a45df8dd9d2b4b12a2bbbd08fa7258d7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/a45df8dd9d2b4b12a2bbbd08fa7258d7 2023-04-19 07:54:14,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/cde6202354964bbeb6fb8579017dcd3e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/cde6202354964bbeb6fb8579017dcd3e 2023-04-19 07:54:14,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/6c87604ada034854b67520f7410d9e75 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/6c87604ada034854b67520f7410d9e75 2023-04-19 07:54:14,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/62e60c92741447c19484dd05b6e3432f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/62e60c92741447c19484dd05b6e3432f 2023-04-19 07:54:14,687 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/fed6e3151db34f48b9c452a5ea87655c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/fed6e3151db34f48b9c452a5ea87655c 2023-04-19 07:54:14,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e8f86f2d016d4524a7dd896277e9c24e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/e8f86f2d016d4524a7dd896277e9c24e 2023-04-19 07:54:14,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/62465e0db055451f90f74f5aacf09261 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/62465e0db055451f90f74f5aacf09261 2023-04-19 07:54:14,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/911f8f07b61f4aa29ce47d31db2164b2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/911f8f07b61f4aa29ce47d31db2164b2 2023-04-19 07:54:14,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/bd1b18ee7f3740dd8977a0cd3f5e21e8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/bd1b18ee7f3740dd8977a0cd3f5e21e8 2023-04-19 07:54:14,692 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/30f9b4b6650d44738cc829c6c80e6d0b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/30f9b4b6650d44738cc829c6c80e6d0b 2023-04-19 07:54:14,693 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/3c2e72e0c6424c129cc38f0a6636298a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/3c2e72e0c6424c129cc38f0a6636298a 2023-04-19 07:54:14,718 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4113b58feb9347faa6002ab984626446, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/46c5a88051be4033874f4210447310e7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/5534937eca1342fa96cf57273b97c327, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/be3abaab292642959b61df83a645afb4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/514bbcf6beed4267b3c876e824b956e0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/3789208365b64fe2b58d2b40f1961e8e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/5bae100107ec4813bcf91b97100e874b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/8ac386c0987a4885a031631f4136bbab, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4a4defb796164216b4aa91a18437962a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/c96e0b27feb54c8586994e93afca2b2c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/714a6ad28fc34152a6893c292b611f87, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/54b37c12de5d4dc8842e1ee72c004e67, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/86729f33b4aa420d8243fc7aa85bf34b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/dabe6f6e73414019b14156e8571a48fb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/05d874b429d443568e5d95aaff0c91fa, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/daad1104dac14329b92b6608fb683217, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/b9eb7b6b08bd4c8c9be5f32f12d9d7c8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/214cf6a0cd7b40fd9ba890457435ccbc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/64ae6381c9994c91b7001d5e1f7023e2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6e952b43ae774b759a0bcdc6c7499213, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/ad99d35691df4158b37872f90bec0406, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/04dfb2fabbfd4d8c82c5f17bc6b76d5e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/38f991b196e347aab26c76e9aa024496, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/f92964c2c94e4440a3ba6f319df52a7a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/aed165a0573448a1853f545edeb74e9c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/678085a753cf48f89bb276bd65ce9be8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/315441ff37fb4a2d9369e1ac9fab16aa, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4640738f587c46ac9f44df9dbcb8f1da, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6c948afba99f4d3ca6ea967d6409dd6e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/588fe2ba7fbc497fbea28d7815dc019a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/a2ebc5faf069435da2b015f5e76e563c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/0e9dfa82356942e28bd797f5b9b8a9d7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/0d1975c9c1a74c4aa9d31cf40c17e0a6, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/cf46a7d2edb54d3faa591281a2b4d61a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6873e1c6674f4c268ad77f41fb4d3b6a] to archive 2023-04-19 07:54:14,719 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:54:14,721 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4113b58feb9347faa6002ab984626446 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4113b58feb9347faa6002ab984626446 2023-04-19 07:54:14,722 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/46c5a88051be4033874f4210447310e7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/46c5a88051be4033874f4210447310e7 2023-04-19 07:54:14,723 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/5534937eca1342fa96cf57273b97c327 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/5534937eca1342fa96cf57273b97c327 2023-04-19 07:54:14,724 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/be3abaab292642959b61df83a645afb4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/be3abaab292642959b61df83a645afb4 2023-04-19 07:54:14,725 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/514bbcf6beed4267b3c876e824b956e0 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/514bbcf6beed4267b3c876e824b956e0 2023-04-19 07:54:14,726 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/3789208365b64fe2b58d2b40f1961e8e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/3789208365b64fe2b58d2b40f1961e8e 2023-04-19 07:54:14,727 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/5bae100107ec4813bcf91b97100e874b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/5bae100107ec4813bcf91b97100e874b 2023-04-19 07:54:14,729 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/8ac386c0987a4885a031631f4136bbab to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/8ac386c0987a4885a031631f4136bbab 2023-04-19 07:54:14,730 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4a4defb796164216b4aa91a18437962a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4a4defb796164216b4aa91a18437962a 2023-04-19 07:54:14,731 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/c96e0b27feb54c8586994e93afca2b2c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/c96e0b27feb54c8586994e93afca2b2c 2023-04-19 07:54:14,733 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/714a6ad28fc34152a6893c292b611f87 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/714a6ad28fc34152a6893c292b611f87 2023-04-19 07:54:14,734 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/54b37c12de5d4dc8842e1ee72c004e67 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/54b37c12de5d4dc8842e1ee72c004e67 2023-04-19 07:54:14,736 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/86729f33b4aa420d8243fc7aa85bf34b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/86729f33b4aa420d8243fc7aa85bf34b 2023-04-19 07:54:14,737 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/dabe6f6e73414019b14156e8571a48fb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/dabe6f6e73414019b14156e8571a48fb 2023-04-19 07:54:14,738 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/05d874b429d443568e5d95aaff0c91fa to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/05d874b429d443568e5d95aaff0c91fa 2023-04-19 07:54:14,740 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/daad1104dac14329b92b6608fb683217 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/daad1104dac14329b92b6608fb683217 2023-04-19 07:54:14,741 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/b9eb7b6b08bd4c8c9be5f32f12d9d7c8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/b9eb7b6b08bd4c8c9be5f32f12d9d7c8 2023-04-19 07:54:14,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/214cf6a0cd7b40fd9ba890457435ccbc to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/214cf6a0cd7b40fd9ba890457435ccbc 2023-04-19 07:54:14,743 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/64ae6381c9994c91b7001d5e1f7023e2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/64ae6381c9994c91b7001d5e1f7023e2 2023-04-19 07:54:14,745 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6e952b43ae774b759a0bcdc6c7499213 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6e952b43ae774b759a0bcdc6c7499213 2023-04-19 07:54:14,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/ad99d35691df4158b37872f90bec0406 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/ad99d35691df4158b37872f90bec0406 2023-04-19 07:54:14,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/04dfb2fabbfd4d8c82c5f17bc6b76d5e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/04dfb2fabbfd4d8c82c5f17bc6b76d5e 2023-04-19 07:54:14,748 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/38f991b196e347aab26c76e9aa024496 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/38f991b196e347aab26c76e9aa024496 2023-04-19 07:54:14,749 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/f92964c2c94e4440a3ba6f319df52a7a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/f92964c2c94e4440a3ba6f319df52a7a 2023-04-19 07:54:14,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/aed165a0573448a1853f545edeb74e9c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/aed165a0573448a1853f545edeb74e9c 2023-04-19 07:54:14,752 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/678085a753cf48f89bb276bd65ce9be8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/678085a753cf48f89bb276bd65ce9be8 2023-04-19 07:54:14,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/315441ff37fb4a2d9369e1ac9fab16aa to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/315441ff37fb4a2d9369e1ac9fab16aa 2023-04-19 07:54:14,755 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4640738f587c46ac9f44df9dbcb8f1da to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/4640738f587c46ac9f44df9dbcb8f1da 2023-04-19 07:54:14,756 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6c948afba99f4d3ca6ea967d6409dd6e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6c948afba99f4d3ca6ea967d6409dd6e 2023-04-19 07:54:14,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/588fe2ba7fbc497fbea28d7815dc019a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/588fe2ba7fbc497fbea28d7815dc019a 2023-04-19 07:54:14,759 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/a2ebc5faf069435da2b015f5e76e563c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/a2ebc5faf069435da2b015f5e76e563c 2023-04-19 07:54:14,760 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/0e9dfa82356942e28bd797f5b9b8a9d7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/0e9dfa82356942e28bd797f5b9b8a9d7 2023-04-19 07:54:14,761 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/0d1975c9c1a74c4aa9d31cf40c17e0a6 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/0d1975c9c1a74c4aa9d31cf40c17e0a6 2023-04-19 07:54:14,763 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/cf46a7d2edb54d3faa591281a2b4d61a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/cf46a7d2edb54d3faa591281a2b4d61a 2023-04-19 07:54:14,764 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6873e1c6674f4c268ad77f41fb4d3b6a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/6873e1c6674f4c268ad77f41fb4d3b6a 2023-04-19 07:54:14,767 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/53c3a03667a247c2a346b7e75bd6c413, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7b65e089aa37451a8fbeb6b6c5e46782, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/96c66aaeaa1f4985a8f4df8ecbf4a37d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0d3df69d957b4e5dbc9862a959495fc8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/e1f665d0703e4a39b73b872c56b0a9ba, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/28c0d9922233432d8b48c6fcbd537a4b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/4562290cf5434c2bbec342791ab1c778, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/ff6a0f92720c4621827579aa3a12e596, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7669c97a72014713a233eefd8ca4b449, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/06525ba9416740038df3865fba66447d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/5e4750d2087c49ce9f930d341d0f54fc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b69a46f736b54f97ad457d8d4b8da88d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7ae649aa61ae44759882e4512d2f51ee, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/6f56d6f92b5e440c9113e348111b670f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/6ff7371e3a4c41ce827bd89c6e39054c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/5bc9d21f07ec44298747cfd87b91e294, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0d022e7c507f42b3b45351edd4216041, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/9590d2647cea429dabc5b06d48f5af85, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/aa005343b54a47b884ef5242840c9cb5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/e15b4cd378724f13b4f9d14e51e69803, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7d2fdb20ed8948d6b6958bd97ae96572, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b60b5afe680d40c791565524274d5df6, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/19096e4b768f4ad6aea590da6c7c0698, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/54e0407fd32144eca85f715990e0501c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/485fb7eba7bd4a79bad392b08d21c24a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/78ae207c29354f039b9fdb4aa7ae6499, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b7c22128a5fb40f29e60d34f2e6e8d6b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/57f770c107dc412e937a528c3351c513, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/673cc4dd3e02417fabdfd01499371639, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/260ca567b16246dbbd29383b73545983, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/64156c60056043c3b367a0fe7e55a2d0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/ac9eea73c7b04feeaa39f62fa13788d4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0585128365db49d9a6356179ca1dfea9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/34f3683733e24f9c9d256f0de26ce466, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/908dd5b4050b410aab11eee79354bc43] to archive 2023-04-19 07:54:14,768 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:54:14,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/53c3a03667a247c2a346b7e75bd6c413 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/53c3a03667a247c2a346b7e75bd6c413 2023-04-19 07:54:14,771 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7b65e089aa37451a8fbeb6b6c5e46782 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7b65e089aa37451a8fbeb6b6c5e46782 2023-04-19 07:54:14,772 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/96c66aaeaa1f4985a8f4df8ecbf4a37d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/96c66aaeaa1f4985a8f4df8ecbf4a37d 2023-04-19 07:54:14,774 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0d3df69d957b4e5dbc9862a959495fc8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0d3df69d957b4e5dbc9862a959495fc8 2023-04-19 07:54:14,775 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/e1f665d0703e4a39b73b872c56b0a9ba to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/e1f665d0703e4a39b73b872c56b0a9ba 2023-04-19 07:54:14,776 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/28c0d9922233432d8b48c6fcbd537a4b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/28c0d9922233432d8b48c6fcbd537a4b 2023-04-19 07:54:14,778 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/4562290cf5434c2bbec342791ab1c778 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/4562290cf5434c2bbec342791ab1c778 2023-04-19 07:54:14,779 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/ff6a0f92720c4621827579aa3a12e596 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/ff6a0f92720c4621827579aa3a12e596 2023-04-19 07:54:14,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7669c97a72014713a233eefd8ca4b449 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7669c97a72014713a233eefd8ca4b449 2023-04-19 07:54:14,781 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/06525ba9416740038df3865fba66447d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/06525ba9416740038df3865fba66447d 2023-04-19 07:54:14,782 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/5e4750d2087c49ce9f930d341d0f54fc to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/5e4750d2087c49ce9f930d341d0f54fc 2023-04-19 07:54:14,783 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b69a46f736b54f97ad457d8d4b8da88d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b69a46f736b54f97ad457d8d4b8da88d 2023-04-19 07:54:14,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7ae649aa61ae44759882e4512d2f51ee to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7ae649aa61ae44759882e4512d2f51ee 2023-04-19 07:54:14,786 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/6f56d6f92b5e440c9113e348111b670f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/6f56d6f92b5e440c9113e348111b670f 2023-04-19 07:54:14,787 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/6ff7371e3a4c41ce827bd89c6e39054c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/6ff7371e3a4c41ce827bd89c6e39054c 2023-04-19 07:54:14,788 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/5bc9d21f07ec44298747cfd87b91e294 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/5bc9d21f07ec44298747cfd87b91e294 2023-04-19 07:54:14,790 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0d022e7c507f42b3b45351edd4216041 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0d022e7c507f42b3b45351edd4216041 2023-04-19 07:54:14,791 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/9590d2647cea429dabc5b06d48f5af85 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/9590d2647cea429dabc5b06d48f5af85 2023-04-19 07:54:14,792 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/aa005343b54a47b884ef5242840c9cb5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/aa005343b54a47b884ef5242840c9cb5 2023-04-19 07:54:14,793 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/e15b4cd378724f13b4f9d14e51e69803 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/e15b4cd378724f13b4f9d14e51e69803 2023-04-19 07:54:14,795 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7d2fdb20ed8948d6b6958bd97ae96572 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/7d2fdb20ed8948d6b6958bd97ae96572 2023-04-19 07:54:14,796 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b60b5afe680d40c791565524274d5df6 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b60b5afe680d40c791565524274d5df6 2023-04-19 07:54:14,797 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/19096e4b768f4ad6aea590da6c7c0698 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/19096e4b768f4ad6aea590da6c7c0698 2023-04-19 07:54:14,798 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/54e0407fd32144eca85f715990e0501c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/54e0407fd32144eca85f715990e0501c 2023-04-19 07:54:14,799 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/485fb7eba7bd4a79bad392b08d21c24a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/485fb7eba7bd4a79bad392b08d21c24a 2023-04-19 07:54:14,800 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/78ae207c29354f039b9fdb4aa7ae6499 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/78ae207c29354f039b9fdb4aa7ae6499 2023-04-19 07:54:14,802 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b7c22128a5fb40f29e60d34f2e6e8d6b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/b7c22128a5fb40f29e60d34f2e6e8d6b 2023-04-19 07:54:14,803 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/57f770c107dc412e937a528c3351c513 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/57f770c107dc412e937a528c3351c513 2023-04-19 07:54:14,804 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/673cc4dd3e02417fabdfd01499371639 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/673cc4dd3e02417fabdfd01499371639 2023-04-19 07:54:14,805 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/260ca567b16246dbbd29383b73545983 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/260ca567b16246dbbd29383b73545983 2023-04-19 07:54:14,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/64156c60056043c3b367a0fe7e55a2d0 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/64156c60056043c3b367a0fe7e55a2d0 2023-04-19 07:54:14,807 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/ac9eea73c7b04feeaa39f62fa13788d4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/ac9eea73c7b04feeaa39f62fa13788d4 2023-04-19 07:54:14,808 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0585128365db49d9a6356179ca1dfea9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/0585128365db49d9a6356179ca1dfea9 2023-04-19 07:54:14,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/34f3683733e24f9c9d256f0de26ce466 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/34f3683733e24f9c9d256f0de26ce466 2023-04-19 07:54:14,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/908dd5b4050b410aab11eee79354bc43 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/908dd5b4050b410aab11eee79354bc43 2023-04-19 07:54:14,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/recovered.edits/765.seqid, newMaxSeqId=765, maxSeqId=1 2023-04-19 07:54:14,816 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71. 2023-04-19 07:54:14,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3c54cc0aa26fee1a2f14917923d1bb71: 2023-04-19 07:54:14,818 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:54:14,818 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=3c54cc0aa26fee1a2f14917923d1bb71, regionState=CLOSED 2023-04-19 07:54:14,819 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890854818"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890854818"}]},"ts":"1681890854818"} 2023-04-19 07:54:14,821 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-19 07:54:14,821 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 3c54cc0aa26fee1a2f14917923d1bb71, server=jenkins-hbase4.apache.org,42359,1681890819191 in 1.6070 sec 2023-04-19 07:54:14,823 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-19 07:54:14,823 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3c54cc0aa26fee1a2f14917923d1bb71, UNASSIGN in 1.6140 sec 2023-04-19 07:54:14,824 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890854823"}]},"ts":"1681890854823"} 2023-04-19 07:54:14,825 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-19 07:54:14,826 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-19 07:54:14,828 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7140 sec 2023-04-19 07:54:15,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-19 07:54:15,232 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-19 07:54:15,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-19 07:54:15,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:15,245 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:15,246 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:15,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-19 07:54:15,251 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:54:15,254 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/recovered.edits] 2023-04-19 07:54:15,257 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/42b382c056fd4a8ab492bd98ccec1822 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/42b382c056fd4a8ab492bd98ccec1822 2023-04-19 07:54:15,258 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/709ef7ae0e744eaebc22bf4243c13cff to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/A/709ef7ae0e744eaebc22bf4243c13cff 2023-04-19 07:54:15,261 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/2cc9ece63af94b20a976ff69e6bd163e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/2cc9ece63af94b20a976ff69e6bd163e 2023-04-19 07:54:15,262 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/e3f932241ef441c0bbf7988d35a65df5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/B/e3f932241ef441c0bbf7988d35a65df5 2023-04-19 07:54:15,265 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/08fe0bebe1f64e7c9ac29d8c49be4bef to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/08fe0bebe1f64e7c9ac29d8c49be4bef 2023-04-19 07:54:15,267 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/12ace067c01d4a7e8405d854c019c76b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/C/12ace067c01d4a7e8405d854c019c76b 2023-04-19 07:54:15,270 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/recovered.edits/765.seqid to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71/recovered.edits/765.seqid 2023-04-19 07:54:15,270 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/3c54cc0aa26fee1a2f14917923d1bb71 2023-04-19 07:54:15,270 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 07:54:15,273 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:15,294 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-19 07:54:15,337 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-19 07:54:15,338 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:15,338 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-19 07:54:15,339 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681890855338"}]},"ts":"9223372036854775807"} 2023-04-19 07:54:15,341 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-19 07:54:15,341 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3c54cc0aa26fee1a2f14917923d1bb71, NAME => 'TestAcidGuarantees,,1681890823236.3c54cc0aa26fee1a2f14917923d1bb71.', STARTKEY => '', ENDKEY => ''}] 2023-04-19 07:54:15,341 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-19 07:54:15,341 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681890855341"}]},"ts":"9223372036854775807"} 2023-04-19 07:54:15,343 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-19 07:54:15,345 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:15,346 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 107 msec 2023-04-19 07:54:15,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-19 07:54:15,349 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-19 07:54:15,365 INFO [Listener at localhost/42517] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=230 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x52d79795-shared-pool-23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:42359-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-189064103_15 at /127.0.0.1:36656 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x52d79795-shared-pool-22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x52d79795-shared-pool-24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x52d79795-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=539 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=424 (was 317) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6408 (was 7126) 2023-04-19 07:54:15,375 INFO [Listener at localhost/42517] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=230, OpenFileDescriptor=539, MaxFileDescriptor=60000, SystemLoadAverage=424, ProcessCount=169, AvailableMemoryMB=6407 2023-04-19 07:54:15,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 07:54:15,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 07:54:15,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:15,380 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-19 07:54:15,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-19 07:54:15,381 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-19 07:54:15,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-19 07:54:15,382 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:15,383 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3 empty. 2023-04-19 07:54:15,383 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:15,383 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 07:54:15,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-19 07:54:15,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-19 07:54:15,792 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 07:54:15,813 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2117921701b6b45f9e01175de52f16a3, NAME => 'TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp 2023-04-19 07:54:15,820 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:54:15,820 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2117921701b6b45f9e01175de52f16a3, disabling compactions & flushes 2023-04-19 07:54:15,820 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:15,820 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:15,820 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. after waiting 0 ms 2023-04-19 07:54:15,820 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:15,820 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:15,820 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:15,822 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-19 07:54:15,823 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890855823"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890855823"}]},"ts":"1681890855823"} 2023-04-19 07:54:15,824 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-19 07:54:15,825 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-19 07:54:15,825 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890855825"}]},"ts":"1681890855825"} 2023-04-19 07:54:15,826 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-19 07:54:15,831 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2117921701b6b45f9e01175de52f16a3, ASSIGN}] 2023-04-19 07:54:15,833 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2117921701b6b45f9e01175de52f16a3, ASSIGN 2023-04-19 07:54:15,833 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2117921701b6b45f9e01175de52f16a3, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42359,1681890819191; forceNewPlan=false, retain=false 2023-04-19 07:54:15,984 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=2117921701b6b45f9e01175de52f16a3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:15,985 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890855984"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890855984"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890855984"}]},"ts":"1681890855984"} 2023-04-19 07:54:15,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-19 07:54:15,986 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:54:16,142 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:16,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2117921701b6b45f9e01175de52f16a3, NAME => 'TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.', STARTKEY => '', ENDKEY => ''} 2023-04-19 07:54:16,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:16,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:54:16,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:16,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:16,144 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:16,146 DEBUG [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A 2023-04-19 07:54:16,146 DEBUG [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A 2023-04-19 07:54:16,146 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:54:16,146 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2117921701b6b45f9e01175de52f16a3 columnFamilyName A 2023-04-19 07:54:16,147 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.HStore(310): Store=2117921701b6b45f9e01175de52f16a3/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:54:16,147 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:16,148 DEBUG [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B 2023-04-19 07:54:16,148 DEBUG [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B 2023-04-19 07:54:16,148 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:54:16,148 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2117921701b6b45f9e01175de52f16a3 columnFamilyName B 2023-04-19 07:54:16,149 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.HStore(310): Store=2117921701b6b45f9e01175de52f16a3/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:54:16,149 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:16,150 DEBUG [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C 2023-04-19 07:54:16,150 DEBUG [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C 2023-04-19 07:54:16,150 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:54:16,150 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2117921701b6b45f9e01175de52f16a3 columnFamilyName C 2023-04-19 07:54:16,151 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.HStore(310): Store=2117921701b6b45f9e01175de52f16a3/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:54:16,151 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:16,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:16,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:16,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 07:54:16,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:16,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 07:54:16,157 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2117921701b6b45f9e01175de52f16a3; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64830483, jitterRate=-0.03395052254199982}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 07:54:16,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:16,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., pid=18, masterSystemTime=1681890856138 2023-04-19 07:54:16,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:16,160 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:16,160 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=2117921701b6b45f9e01175de52f16a3, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:16,161 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890856160"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681890856160"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681890856160"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681890856160"}]},"ts":"1681890856160"} 2023-04-19 07:54:16,164 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-19 07:54:16,164 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 in 176 msec 2023-04-19 07:54:16,166 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-19 07:54:16,166 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2117921701b6b45f9e01175de52f16a3, ASSIGN in 333 msec 2023-04-19 07:54:16,167 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-19 07:54:16,167 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890856167"}]},"ts":"1681890856167"} 2023-04-19 07:54:16,168 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-19 07:54:16,170 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-19 07:54:16,171 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 792 msec 2023-04-19 07:54:16,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-19 07:54:16,486 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-19 07:54:16,487 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x0a607873 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:16,493 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31436cf6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:16,494 DEBUG [Listener at localhost/42517] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:16,498 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54006, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:16,500 DEBUG [Listener at localhost/42517] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-19 07:54:16,503 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40868, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-19 07:54:16,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 07:54:16,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 07:54:16,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:16,935 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-19 07:54:16,936 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 07:54:16,936 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-19 07:54:16,940 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-19 07:54:16,952 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2117921701b6b45f9e01175de52f16a3, REOPEN/MOVE}] 2023-04-19 07:54:16,953 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2117921701b6b45f9e01175de52f16a3, REOPEN/MOVE 2023-04-19 07:54:16,954 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=2117921701b6b45f9e01175de52f16a3, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:16,954 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890856954"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890856954"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890856954"}]},"ts":"1681890856954"} 2023-04-19 07:54:16,955 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:54:17,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:17,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2117921701b6b45f9e01175de52f16a3, disabling compactions & flushes 2023-04-19 07:54:17,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:17,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:17,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. after waiting 0 ms 2023-04-19 07:54:17,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:17,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-19 07:54:17,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:17,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:17,113 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 2117921701b6b45f9e01175de52f16a3 to self. 2023-04-19 07:54:17,114 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:17,114 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=2117921701b6b45f9e01175de52f16a3, regionState=CLOSED 2023-04-19 07:54:17,115 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890857114"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890857114"}]},"ts":"1681890857114"} 2023-04-19 07:54:17,117 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-19 07:54:17,117 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 in 161 msec 2023-04-19 07:54:17,118 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2117921701b6b45f9e01175de52f16a3, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,42359,1681890819191; forceNewPlan=false, retain=true 2023-04-19 07:54:17,268 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=2117921701b6b45f9e01175de52f16a3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,268 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890857268"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890857268"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890857268"}]},"ts":"1681890857268"} 2023-04-19 07:54:17,270 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:54:17,425 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:17,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2117921701b6b45f9e01175de52f16a3, NAME => 'TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.', STARTKEY => '', ENDKEY => ''} 2023-04-19 07:54:17,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:17,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:54:17,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:17,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:17,428 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:17,429 DEBUG [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A 2023-04-19 07:54:17,429 DEBUG [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A 2023-04-19 07:54:17,430 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:54:17,432 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2117921701b6b45f9e01175de52f16a3 columnFamilyName A 2023-04-19 07:54:17,434 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.HStore(310): Store=2117921701b6b45f9e01175de52f16a3/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:54:17,435 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:17,436 DEBUG [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B 2023-04-19 07:54:17,436 DEBUG [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B 2023-04-19 07:54:17,436 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:54:17,437 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2117921701b6b45f9e01175de52f16a3 columnFamilyName B 2023-04-19 07:54:17,437 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.HStore(310): Store=2117921701b6b45f9e01175de52f16a3/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:54:17,437 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:17,438 DEBUG [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C 2023-04-19 07:54:17,438 DEBUG [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C 2023-04-19 07:54:17,438 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:54:17,439 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2117921701b6b45f9e01175de52f16a3 columnFamilyName C 2023-04-19 07:54:17,439 INFO [StoreOpener-2117921701b6b45f9e01175de52f16a3-1] regionserver.HStore(310): Store=2117921701b6b45f9e01175de52f16a3/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:54:17,439 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:17,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:17,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:17,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 07:54:17,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:17,445 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2117921701b6b45f9e01175de52f16a3; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66470462, jitterRate=-0.009512931108474731}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 07:54:17,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:17,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., pid=23, masterSystemTime=1681890857422 2023-04-19 07:54:17,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:17,447 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:17,448 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=2117921701b6b45f9e01175de52f16a3, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,448 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890857448"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681890857448"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681890857448"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681890857448"}]},"ts":"1681890857448"} 2023-04-19 07:54:17,451 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-19 07:54:17,451 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 in 179 msec 2023-04-19 07:54:17,452 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-19 07:54:17,452 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2117921701b6b45f9e01175de52f16a3, REOPEN/MOVE in 499 msec 2023-04-19 07:54:17,455 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-19 07:54:17,455 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 514 msec 2023-04-19 07:54:17,457 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 941 msec 2023-04-19 07:54:17,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-19 07:54:17,466 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x399105f2 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:17,471 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b3934fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:17,471 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x55cbb115 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:17,476 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bbdba17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:17,476 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x0a71edf0 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:17,483 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2bab6b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:17,483 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x24bd3872 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:17,490 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73e51571, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:17,490 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x7aba8e46 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:17,494 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@370947cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:17,494 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x210d1008 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:17,498 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c472cfd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:17,499 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x00c2598b to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:17,502 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@498694cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:17,503 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x508246cc to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:17,508 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@421e5356, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:17,508 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x08795451 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:17,515 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a0fd7b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:17,525 DEBUG [hconnection-0x29887de-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:17,526 DEBUG [hconnection-0x53bc75ae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:17,526 DEBUG [hconnection-0x75fd8bb4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:17,526 DEBUG [hconnection-0x7c6efcd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:17,531 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54018, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:17,531 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:17,531 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54042, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:17,532 DEBUG [hconnection-0x3027d585-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:17,532 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54044, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:17,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:17,533 DEBUG [hconnection-0x3348937b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:17,533 DEBUG [hconnection-0x2f78307e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:17,535 DEBUG [hconnection-0x6f4398b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:17,536 DEBUG [hconnection-0x6c4cf999-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:17,536 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54058, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:17,536 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54054, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:17,536 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:17,543 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54064, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:17,543 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:17,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:17,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:17,547 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:17,547 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:17,547 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:17,547 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:17,549 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:17,549 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:17,552 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,552 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:17,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:17,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:17,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:17,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:17,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:17,554 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:17,554 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:17,554 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:17,554 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:17,554 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:17,554 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:17,554 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:17,555 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 07:54:17,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:17,556 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:17,556 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,556 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:17,556 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:17,557 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:17,557 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:17,557 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:17,557 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:17,557 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:17,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890917572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890857570, max timestamp=1681890857572Num uniques 859; 2023-04-19 07:54:17,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:17,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890857569, max timestamp=1681890857572Num uniques 859; 2023-04-19 07:54:17,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:54:17,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681890857570, max timestamp=1681890857572Num uniques 859; 2023-04-19 07:54:17,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-19 07:54:17,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890917597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890917600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54044 deadline: 1681890917602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54054 deadline: 1681890917602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,632 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304194198a056d9bb41088c527bf0c86bf5d2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194198a056d9bb41088c527bf0c86bf5d2 2023-04-19 07:54:17,659 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/b896c9e93e1d4b8e8704fe2dc76a142b 2023-04-19 07:54:17,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890917679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890917704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890917705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54044 deadline: 1681890917708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54054 deadline: 1681890917710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890917885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890917906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890917907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54044 deadline: 1681890917911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:17,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:17,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54054 deadline: 1681890917913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,118 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/2fc3305f66c043af8312caf309122a2f 2023-04-19 07:54:18,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:18,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890918189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:18,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890918212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:18,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890918213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:18,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54044 deadline: 1681890918215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:18,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54054 deadline: 1681890918217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,538 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/7afd3c7c092e4135bc7e7e98aa804f45 2023-04-19 07:54:18,546 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/b896c9e93e1d4b8e8704fe2dc76a142b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/b896c9e93e1d4b8e8704fe2dc76a142b 2023-04-19 07:54:18,553 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/b896c9e93e1d4b8e8704fe2dc76a142b, entries=200, sequenceid=15, filesize=31.9 K 2023-04-19 07:54:18,554 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/2fc3305f66c043af8312caf309122a2f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fc3305f66c043af8312caf309122a2f 2023-04-19 07:54:18,561 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fc3305f66c043af8312caf309122a2f, entries=150, sequenceid=15, filesize=11.6 K 2023-04-19 07:54:18,562 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/7afd3c7c092e4135bc7e7e98aa804f45 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/7afd3c7c092e4135bc7e7e98aa804f45 2023-04-19 07:54:18,567 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/7afd3c7c092e4135bc7e7e98aa804f45, entries=150, sequenceid=15, filesize=11.6 K 2023-04-19 07:54:18,568 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=124.84 KB/127835 for 2117921701b6b45f9e01175de52f16a3 in 1013ms, sequenceid=15, compaction requested=false 2023-04-19 07:54:18,568 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-19 07:54:18,571 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:18,571 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:18,571 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:18,571 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:18,571 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,571 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:18,571 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:18,573 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,573 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:18,574 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,574 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:18,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:18,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22c72287[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:18,575 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:18,575 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,577 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:18,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:18,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:18,578 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,578 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:18,579 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,579 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:18,580 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:18,580 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:18,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:18,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:18,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:18,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:18,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:18,583 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:18,583 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:18,583 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:18,583 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:18,583 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:18,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:18,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:18,586 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:18,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,588 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:18,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,591 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:18,591 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,591 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:18,591 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:18,591 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:18,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:18,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:18,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:18,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:18,591 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:18,591 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,591 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,591 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,591 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:18,591 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:18,592 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:18,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:18,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:18,692 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:18,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:18,693 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=131.55 KB heapSize=374.30 KB 2023-04-19 07:54:18,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:18,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:18,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:18,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:18,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:18,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:18,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:18,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:18,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:18,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:18,715 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:18,715 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:18,715 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:18,715 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:18,716 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,716 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:18,718 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,719 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:18,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:18,719 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,719 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:18,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:18,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:18,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890918718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:18,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:18,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890918718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:18,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:18,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:18,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890918719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:18,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54054 deadline: 1681890918721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:18,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:18,725 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:18,725 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:18,725 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:18,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:18,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:18,726 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:18,726 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:18,727 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:18,727 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:18,727 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:18,727 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:18,727 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,727 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:18,727 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:18,729 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:18,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:18,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:18,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54044 deadline: 1681890918723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:18,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:18,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:18,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:18,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@495bdac0[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:18,733 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:18,733 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,734 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,735 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,735 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:18,736 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,736 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:18,736 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:18,736 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:18,736 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:18,737 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:18,737 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:18,737 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:18,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:18,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:18,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:18,737 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:18,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:18,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:18,741 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:18,741 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:18,741 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:18,741 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:18,741 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:18,743 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:18,744 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,744 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:18,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,744 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:18,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:18,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:18,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:18,746 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:18,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:18,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:18,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:18,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:18,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:18,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:18,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:18,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:18,750 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:18,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,751 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,751 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,751 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:18,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:18,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:18,751 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:18,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:18,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:18,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890918823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,851 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:18,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:18,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:18,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:18,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:18,860 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:18,860 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:18,860 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:18,860 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:18,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:18,863 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:18,863 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:18,864 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,864 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:18,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:18,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:18,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:18,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:18,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:18,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:18,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:18,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:18,867 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:18,867 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:18,867 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:18,867 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:18,868 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:18,868 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:18,868 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:18,868 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:18,868 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,868 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:18,868 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:18,873 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,873 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:18,873 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,873 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:18,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:18,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ecf08cc[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:18,875 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:18,876 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,877 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,877 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:18,877 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:18,877 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,877 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:18,877 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:18,879 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:18,879 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:18,879 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:18,879 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:18,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:18,883 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:18,883 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:18,883 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:18,883 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:18,883 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:18,885 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,885 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:18,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,885 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:18,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:18,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:18,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:18,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:18,889 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:18,890 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:18,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,895 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,895 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:18,895 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:18,895 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:18,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:18,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:18,895 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:18,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:18,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:18,895 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:18,895 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,895 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:18,895 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:18,895 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:18,895 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:18,896 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:18,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:18,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:18,996 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:18,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:18,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:19,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:19,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:19,001 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:19,001 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,001 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:19,001 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:19,001 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,001 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,002 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,002 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,003 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,003 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:19,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:19,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:19,008 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:19,008 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,008 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:19,012 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:19,012 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:19,012 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,012 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:19,012 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:19,012 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,012 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:19,013 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:19,013 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,013 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:19,013 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:19,014 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,015 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,015 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,016 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:19,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@918abd5[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:19,017 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:19,017 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,018 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,018 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,018 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:19,018 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:19,019 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:19,019 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:19,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:19,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,021 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:19,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:19,024 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:19,024 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:19,024 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:19,024 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:19,024 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:19,026 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,026 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,026 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,027 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,028 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,030 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,030 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,030 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,031 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,031 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,031 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,031 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,031 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,031 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,031 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:19,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:19,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:19,031 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:19,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890919029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,031 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:19,132 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:19,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:19,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:19,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:19,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:19,137 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:19,137 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,137 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:19,137 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:19,137 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,137 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,138 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,139 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,140 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,140 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:19,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:19,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:19,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:19,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:19,144 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:19,144 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:19,144 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,144 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:19,144 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:19,144 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,144 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:19,144 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:19,144 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,144 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:19,144 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:19,147 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,147 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:19,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1250736b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:19,152 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:19,152 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,153 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304197d53b0e81426495b92d39961241fc5c8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197d53b0e81426495b92d39961241fc5c8 2023-04-19 07:54:19,154 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,154 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:19,154 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:19,154 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:19,155 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,155 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:19,158 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:19,158 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,158 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:19,158 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:19,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:19,162 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:19,162 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:19,162 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:19,162 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:19,162 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:19,164 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,164 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,166 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,167 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,169 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,174 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,174 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,175 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,175 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,175 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,175 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,175 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,175 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,175 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,175 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:19,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:19,176 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:19,176 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:19,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,276 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:19,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:19,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:19,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:19,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:19,281 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:19,281 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,281 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:19,281 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:19,281 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,281 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,282 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,282 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,283 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,283 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:19,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:19,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:19,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:19,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:19,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:19,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:19,284 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,284 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:19,284 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:19,284 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:19,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:19,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:19,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,286 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:19,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:19,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ca263b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:19,288 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:19,288 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,289 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,289 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:19,289 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:19,289 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:19,289 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,289 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:19,294 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,294 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:19,294 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,294 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:19,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:19,296 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:19,296 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:19,296 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:19,296 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:19,296 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:19,298 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,298 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,298 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,298 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,299 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:19,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:19,302 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,303 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:19,303 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:19,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:19,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890919335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,403 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:19,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:19,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:19,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:19,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:19,407 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:19,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:19,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:19,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,408 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,409 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,409 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,409 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:19,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:19,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:19,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:19,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:19,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:19,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:19,411 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,411 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:19,411 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:19,411 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:19,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:19,412 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,412 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:19,412 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:19,414 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,414 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,414 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,414 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:19,416 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:19,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33f5c726[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:19,416 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,417 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,417 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:19,417 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:19,417 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:19,417 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,417 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:19,418 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:19,419 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,419 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:19,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:19,422 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:19,422 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:19,422 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:19,422 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:19,422 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:19,424 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,424 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,424 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,425 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,426 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,429 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,429 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,430 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,430 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,430 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:19,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,430 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,430 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,430 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,430 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,430 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,430 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:19,430 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:19,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,530 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:19,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:19,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:19,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:19,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:19,534 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:19,534 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,534 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:19,534 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:19,535 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,535 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,536 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,536 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,536 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:19,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:19,538 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:19,538 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,538 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:19,538 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:19,538 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:19,538 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,538 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:19,538 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:19,538 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,538 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:19,538 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:19,538 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,538 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:19,538 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:19,540 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,540 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,540 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,540 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:19,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fff948e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:19,543 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:19,543 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,544 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,544 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:19,544 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:19,544 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:19,544 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,544 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:19,547 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,547 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,547 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,547 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:19,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:19,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:19,550 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:19,550 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:19,550 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:19,550 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:19,550 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:19,552 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,552 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,552 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,554 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,556 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,567 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:19,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:19,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:19,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,568 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:19,568 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:19,583 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/41f330c6ecc54eafa3316f88fff75cf7 2023-04-19 07:54:19,604 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/dd1b9e6571fa4d58a0a8ea9bb87db77b 2023-04-19 07:54:19,668 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:19,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:19,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:19,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:19,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:19,677 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:19,678 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,678 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:19,678 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:19,679 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,680 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,681 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,681 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,681 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,681 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:19,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:19,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:19,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:19,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:19,683 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:19,683 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,683 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:19,684 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,684 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:19,684 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:19,684 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:19,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,689 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,689 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,689 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ea56b31[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:19,691 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:19,691 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,692 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:19,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:19,692 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,693 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:19,694 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:19,694 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,694 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:19,694 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:19,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:19,699 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:19,699 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:19,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:19,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:19,700 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:19,705 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,705 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,705 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,708 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,714 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,714 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,714 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,714 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,714 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,714 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:19,714 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:19,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:19,714 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,714 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,714 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,715 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:19,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,715 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:19,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890919727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:19,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890919729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:19,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54054 deadline: 1681890919732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:19,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54044 deadline: 1681890919738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,815 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:19,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:19,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:19,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:19,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:19,823 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:19,823 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,823 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:19,823 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:19,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,825 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,825 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,825 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,825 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:19,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:19,835 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:19,835 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,835 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:19,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:19,838 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:19,838 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:19,838 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890919838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,839 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:19,839 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:19,839 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,840 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:19,840 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:19,841 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,843 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:19,844 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:19,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a0a6320[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:19,844 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,846 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,846 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:19,846 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:19,846 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:19,846 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,846 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:19,847 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:19,847 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,847 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:19,847 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:19,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:19,852 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:19,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:19,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:19,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:19,852 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:19,854 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,854 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,854 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,856 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,859 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,862 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,862 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:19,862 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:19,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:19,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,862 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,863 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:19,863 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:19,863 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,863 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,863 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,863 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,863 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,963 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:19,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:19,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:19,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:19,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:19,974 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:19,974 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,974 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:19,974 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:19,974 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,974 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,976 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,976 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,977 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,977 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:19,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:19,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:19,978 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:19,978 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:19,978 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:19,978 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:19,978 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:19,978 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,978 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:19,979 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:19,979 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:19,979 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:19,979 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:19,979 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,979 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:19,979 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:19,980 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,980 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,981 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,981 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:19,982 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:19,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27db5fb0[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:19,982 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,983 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,983 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:19,983 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:19,983 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:19,984 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,984 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:19,985 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,985 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:19,985 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,985 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:19,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:19,988 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:19,988 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:19,988 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:19,988 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:19,988 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:19,990 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,990 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,990 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:19,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:19,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,991 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,993 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,995 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,995 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,995 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:19,995 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:19,995 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:19,995 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:19,995 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:19,995 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:19,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:19,996 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:19,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:19,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:19,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:19,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:19,996 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:19,996 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:20,027 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/3372db3442c740e38ef8c7af0ab12da2 2023-04-19 07:54:20,034 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/41f330c6ecc54eafa3316f88fff75cf7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/41f330c6ecc54eafa3316f88fff75cf7 2023-04-19 07:54:20,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/41f330c6ecc54eafa3316f88fff75cf7, entries=200, sequenceid=51, filesize=31.9 K 2023-04-19 07:54:20,051 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/dd1b9e6571fa4d58a0a8ea9bb87db77b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/dd1b9e6571fa4d58a0a8ea9bb87db77b 2023-04-19 07:54:20,057 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/dd1b9e6571fa4d58a0a8ea9bb87db77b, entries=150, sequenceid=51, filesize=11.6 K 2023-04-19 07:54:20,059 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/3372db3442c740e38ef8c7af0ab12da2 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/3372db3442c740e38ef8c7af0ab12da2 2023-04-19 07:54:20,064 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/3372db3442c740e38ef8c7af0ab12da2, entries=150, sequenceid=51, filesize=11.6 K 2023-04-19 07:54:20,065 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~131.55 KB/134705, heapSize ~374.26 KB/383240, currentSize=46.96 KB/48090 for 2117921701b6b45f9e01175de52f16a3 in 1372ms, sequenceid=51, compaction requested=false 2023-04-19 07:54:20,065 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:20,096 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:20,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:20,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:20,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:20,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:20,101 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:20,101 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:20,101 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:20,101 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:20,103 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:20,103 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:20,104 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:20,104 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:20,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:20,105 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:20,105 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:20,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:20,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:20,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:20,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:20,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:20,107 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:20,107 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:20,107 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:20,108 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:20,108 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:20,108 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:20,108 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:20,108 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:54:20,108 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:20,108 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:20,109 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:20,109 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:20,109 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:20,110 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:20,151 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304190130c31e07f04009b1b8bc1b90f03e2b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304190130c31e07f04009b1b8bc1b90f03e2b 2023-04-19 07:54:20,370 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-19 07:54:20,561 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/69f35e41d5f0421ebc5e7c10d860b34a 2023-04-19 07:54:20,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:20,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:20,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890860869, max timestamp=1681890860893Num uniques 859; 2023-04-19 07:54:20,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:54:20,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890860869, max timestamp=1681890860893Num uniques 859; 2023-04-19 07:54:20,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890860869, max timestamp=1681890860893Num uniques 859; 2023-04-19 07:54:20,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:54:20,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:54:20,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:20,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890920951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:20,973 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/2fc3e690b1b74c75831ecbbbf3827abd 2023-04-19 07:54:21,001 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/015d251b108d49d9983507072b0b3e0c 2023-04-19 07:54:21,006 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/69f35e41d5f0421ebc5e7c10d860b34a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/69f35e41d5f0421ebc5e7c10d860b34a 2023-04-19 07:54:21,012 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/69f35e41d5f0421ebc5e7c10d860b34a, entries=150, sequenceid=61, filesize=25.1 K 2023-04-19 07:54:21,013 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/2fc3e690b1b74c75831ecbbbf3827abd as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fc3e690b1b74c75831ecbbbf3827abd 2023-04-19 07:54:21,018 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fc3e690b1b74c75831ecbbbf3827abd, entries=150, sequenceid=61, filesize=11.6 K 2023-04-19 07:54:21,020 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/015d251b108d49d9983507072b0b3e0c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/015d251b108d49d9983507072b0b3e0c 2023-04-19 07:54:21,028 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/015d251b108d49d9983507072b0b3e0c, entries=150, sequenceid=61, filesize=11.6 K 2023-04-19 07:54:21,029 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2117921701b6b45f9e01175de52f16a3 in 921ms, sequenceid=61, compaction requested=true 2023-04-19 07:54:21,029 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:21,029 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,029 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:21,029 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:21,029 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,029 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:21,029 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:21,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,032 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,032 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,032 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:21,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2483e421[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:21,034 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:21,035 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,037 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,037 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:21,037 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:21,037 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:21,038 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,038 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:21,039 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:21,039 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,039 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,039 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:21,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:21,043 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:21,043 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:21,043 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:21,043 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:21,043 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:21,046 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,046 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,046 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,049 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,052 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,052 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:21,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:21,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,052 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,052 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,053 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,053 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,053 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,053 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,053 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:21,053 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:21,053 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,053 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:21,059 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-19 07:54:21,059 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:21,059 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:21,059 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:21,059 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:21,059 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:21,059 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:21,079 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304192671d145659542f4964b4b28d3e410f8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192671d145659542f4964b4b28d3e410f8 2023-04-19 07:54:21,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:21,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890921090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,153 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:21,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:21,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:21,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:21,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:21,157 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:21,157 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:21,157 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:21,157 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:21,158 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,158 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,159 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,159 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,159 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,159 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:21,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:21,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:21,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:21,160 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:21,161 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:21,161 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:21,161 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:21,161 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:21,161 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,161 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:21,161 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:21,161 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,161 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:21,161 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:21,161 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,161 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:21,161 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:21,162 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,162 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,163 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,163 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:21,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@613aff63[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:21,165 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:21,165 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,166 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,166 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:21,166 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:21,166 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:21,167 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,167 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:21,169 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:21,169 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,169 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,169 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:21,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:21,172 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:21,172 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:21,172 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:21,172 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:21,172 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:21,173 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,173 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,173 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,174 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,175 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,177 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,177 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,177 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,177 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,177 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,177 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,177 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,177 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,177 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:21,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:21,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,177 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,178 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:21,178 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:21,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:21,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890921196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,278 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:21,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:21,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:21,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:21,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:21,282 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:21,282 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:21,282 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:21,282 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:21,282 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,282 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,284 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,284 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,284 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:21,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:21,285 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:21,285 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:21,285 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:21,286 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,286 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:21,286 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:21,286 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:21,288 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,288 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,289 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,289 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:21,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8408307[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:21,290 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:21,290 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,292 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:21,292 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:21,292 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,292 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:21,292 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,293 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,293 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:21,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:21,296 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:21,296 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:21,296 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:21,296 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:21,296 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:21,297 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,297 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,297 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,298 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,299 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,303 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,303 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,303 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,303 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:21,303 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:21,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:21,303 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,303 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,303 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,303 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,303 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:21,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,303 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:21,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890921399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,403 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:21,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:21,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:21,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:21,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:21,407 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:21,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:21,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:21,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:21,408 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,408 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,409 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,409 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,409 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:21,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:21,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:21,410 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:21,411 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,411 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:21,411 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:21,411 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:21,413 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,413 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,413 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,413 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:21,414 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:21,415 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@759b7b60[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:21,415 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:21,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:21,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:21,416 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,416 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:21,418 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:21,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:21,421 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:21,421 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:21,421 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:21,421 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:21,421 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:21,422 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,422 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,422 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,423 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,424 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,429 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,429 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,429 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:21,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:21,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,429 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:21,429 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:21,487 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/d8c331f744e547f9b5f6378b5a826512 2023-04-19 07:54:21,529 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:21,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:21,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:21,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:21,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:21,533 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:21,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:21,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:21,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:21,534 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,534 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,536 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,536 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,536 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:21,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:21,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:21,537 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:21,538 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:21,538 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:21,539 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:21,539 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:21,539 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,539 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:21,539 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:21,539 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,539 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:21,539 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:21,539 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,539 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:21,539 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:21,540 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,540 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,540 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,541 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:21,542 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:21,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e022b1e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:21,543 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,543 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,544 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:21,544 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:21,544 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:21,544 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,544 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:21,545 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,545 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:21,545 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,545 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:21,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:21,547 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:21,547 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:21,547 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:21,547 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:21,547 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:21,550 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,550 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,550 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,551 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,552 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,555 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,555 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,555 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:21,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:21,555 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,555 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,555 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:21,555 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,555 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:21,555 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,555 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,555 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,555 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,655 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:21,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:21,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:21,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:21,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:21,659 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:21,659 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:21,659 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:21,659 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:21,660 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,660 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,661 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,661 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,661 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,662 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:21,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:21,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:21,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:21,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:21,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:21,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:21,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:21,663 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,663 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:21,663 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:21,663 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:21,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:21,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:21,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:21,665 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,665 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,665 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,665 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:21,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7875145a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:21,667 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:21,667 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,669 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:21,669 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,669 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:21,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:21,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:21,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:21,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,670 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:21,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:21,673 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:21,673 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:21,673 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:21,673 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:21,673 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:21,674 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,674 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,674 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,675 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,676 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,679 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:21,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:21,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:21,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,679 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:21,679 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:21,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:21,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890921702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:21,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890921740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,743 DEBUG [Thread-861] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:21,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:21,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890921744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,752 DEBUG [Thread-865] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:21,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:21,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54044 deadline: 1681890921752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,754 DEBUG [Thread-863] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:21,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:21,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54054 deadline: 1681890921755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,757 DEBUG [Thread-859] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:21,780 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:21,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:21,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:21,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:21,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:21,783 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:21,783 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:21,783 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:21,783 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:21,784 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,784 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,786 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,786 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,787 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,787 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:21,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:21,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:21,788 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,788 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:21,788 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:21,788 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:21,790 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,790 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,790 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,790 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:21,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7421f8e8[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:21,791 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:21,792 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,792 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,793 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,793 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:21,794 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,794 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:21,794 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,794 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:21,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:21,801 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:21,802 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:21,802 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:21,802 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:21,802 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:21,804 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,804 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,805 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,806 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,808 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,808 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,808 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,809 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,809 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:21,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:21,809 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,809 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,809 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,809 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,809 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:21,809 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:21,902 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/2fbba6039d094038b2104091e96e6aaf 2023-04-19 07:54:21,909 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:21,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:21,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:21,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:21,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:21,913 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:21,913 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:21,913 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:21,913 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:21,914 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,914 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,915 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/cea8ff20c43c4a919aa85167c6a01a35 2023-04-19 07:54:21,916 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,916 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,916 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,916 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:21,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:21,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:21,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:21,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:21,917 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:21,917 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:21,918 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,918 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:21,918 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:21,918 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:21,921 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/d8c331f744e547f9b5f6378b5a826512 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/d8c331f744e547f9b5f6378b5a826512 2023-04-19 07:54:21,923 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:21,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a4fcd77[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:21,926 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:21,926 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/d8c331f744e547f9b5f6378b5a826512, entries=200, sequenceid=98, filesize=31.9 K 2023-04-19 07:54:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:21,927 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,927 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:21,927 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:21,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/2fbba6039d094038b2104091e96e6aaf as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fbba6039d094038b2104091e96e6aaf 2023-04-19 07:54:21,928 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:21,929 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,929 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:21,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:21,932 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:21,932 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:21,932 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:21,932 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:21,932 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:21,933 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fbba6039d094038b2104091e96e6aaf, entries=150, sequenceid=98, filesize=11.6 K 2023-04-19 07:54:21,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/cea8ff20c43c4a919aa85167c6a01a35 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/cea8ff20c43c4a919aa85167c6a01a35 2023-04-19 07:54:21,935 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,936 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:21,936 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:21,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,937 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,938 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:21,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:21,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/cea8ff20c43c4a919aa85167c6a01a35, entries=150, sequenceid=98, filesize=11.6 K 2023-04-19 07:54:21,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 2117921701b6b45f9e01175de52f16a3 in 881ms, sequenceid=98, compaction requested=true 2023-04-19 07:54:21,940 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:21,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:21,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:21,940 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:21,940 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:21,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:21,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:21,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:21,942 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 123804 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:21,942 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/A is initiating minor compaction (all files) 2023-04-19 07:54:21,942 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/A in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,942 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/b896c9e93e1d4b8e8704fe2dc76a142b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/41f330c6ecc54eafa3316f88fff75cf7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/69f35e41d5f0421ebc5e7c10d860b34a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/d8c331f744e547f9b5f6378b5a826512] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=120.9 K 2023-04-19 07:54:21,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:21,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:21,943 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:21,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:21,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:21,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:21,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:21,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:21,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:21,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:21,943 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:21,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:21,943 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:21,944 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting b896c9e93e1d4b8e8704fe2dc76a142b, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681890857540 2023-04-19 07:54:21,944 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:21,944 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/B is initiating minor compaction (all files) 2023-04-19 07:54:21,944 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/B in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,944 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fc3305f66c043af8312caf309122a2f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/dd1b9e6571fa4d58a0a8ea9bb87db77b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fc3e690b1b74c75831ecbbbf3827abd, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fbba6039d094038b2104091e96e6aaf] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=46.4 K 2023-04-19 07:54:21,946 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 41f330c6ecc54eafa3316f88fff75cf7, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681890857591 2023-04-19 07:54:21,946 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 69f35e41d5f0421ebc5e7c10d860b34a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681890858701 2023-04-19 07:54:21,947 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting d8c331f744e547f9b5f6378b5a826512, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681890860942 2023-04-19 07:54:21,956 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 2fc3305f66c043af8312caf309122a2f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681890857540 2023-04-19 07:54:21,957 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting dd1b9e6571fa4d58a0a8ea9bb87db77b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681890857599 2023-04-19 07:54:21,957 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 2fc3e690b1b74c75831ecbbbf3827abd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681890858701 2023-04-19 07:54:21,958 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 2fbba6039d094038b2104091e96e6aaf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681890860945 2023-04-19 07:54:21,970 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#B#compaction#124 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:21,971 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#A#compaction#123 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:21,988 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/2c88b8d835f44af882a160638ab50d99 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2c88b8d835f44af882a160638ab50d99 2023-04-19 07:54:21,994 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/B of 2117921701b6b45f9e01175de52f16a3 into 2c88b8d835f44af882a160638ab50d99(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:21,994 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:21,994 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/B, priority=12, startTime=1681890861940; duration=0sec 2023-04-19 07:54:21,995 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:21,995 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:21,996 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:21,996 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/C is initiating minor compaction (all files) 2023-04-19 07:54:21,997 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/C in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:21,997 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/7afd3c7c092e4135bc7e7e98aa804f45, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/3372db3442c740e38ef8c7af0ab12da2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/015d251b108d49d9983507072b0b3e0c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/cea8ff20c43c4a919aa85167c6a01a35] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=46.4 K 2023-04-19 07:54:21,997 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7afd3c7c092e4135bc7e7e98aa804f45, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681890857540 2023-04-19 07:54:21,998 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 3372db3442c740e38ef8c7af0ab12da2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681890857599 2023-04-19 07:54:21,998 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 015d251b108d49d9983507072b0b3e0c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681890858701 2023-04-19 07:54:21,999 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting cea8ff20c43c4a919aa85167c6a01a35, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681890860945 2023-04-19 07:54:22,010 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#C#compaction#125 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:22,015 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/a36eace486cb44a8a90f6af9050f5b4d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a36eace486cb44a8a90f6af9050f5b4d 2023-04-19 07:54:22,022 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/A of 2117921701b6b45f9e01175de52f16a3 into a36eace486cb44a8a90f6af9050f5b4d(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:22,022 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:22,022 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/A, priority=12, startTime=1681890861940; duration=0sec 2023-04-19 07:54:22,023 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:22,044 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:22,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:22,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:22,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:22,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:22,047 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:22,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:22,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:22,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:22,048 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,048 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,049 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:22,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:22,049 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,050 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,050 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:22,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:22,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:22,051 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:22,051 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:22,051 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:22,052 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:22,052 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:22,052 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:22,052 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:22,052 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 07:54:22,052 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:22,052 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:22,053 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:22,053 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:22,053 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:22,053 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:22,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:22,207 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:22,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890862236, max timestamp=1681890862250Num uniques 859; 2023-04-19 07:54:22,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890862236, max timestamp=1681890862250Num uniques 859; 2023-04-19 07:54:22,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:22,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890862236, max timestamp=1681890862250Num uniques 859; 2023-04-19 07:54:22,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:22,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:22,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:22,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54052 deadline: 1681890922310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890862294, max timestamp=1681890862309Num uniques 1018; 2023-04-19 07:54:22,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:54:22,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890862294, max timestamp=1681890862309Num uniques 1018; 2023-04-19 07:54:22,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:54:22,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890862294, max timestamp=1681890862309Num uniques 1018; 2023-04-19 07:54:22,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:54:22,442 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/de181201bf6e435689c0335a2290565b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/de181201bf6e435689c0335a2290565b 2023-04-19 07:54:22,456 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/C of 2117921701b6b45f9e01175de52f16a3 into de181201bf6e435689c0335a2290565b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:22,456 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:22,456 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/C, priority=12, startTime=1681890861940; duration=0sec 2023-04-19 07:54:22,456 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:22,474 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041904b269ba74284952af1c96db57084c03 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041904b269ba74284952af1c96db57084c03 2023-04-19 07:54:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890862464, max timestamp=1681890862484Num uniques 1027; 2023-04-19 07:54:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890862464, max timestamp=1681890862484Num uniques 1027; 2023-04-19 07:54:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-19 07:54:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-19 07:54:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890862464, max timestamp=1681890862484Num uniques 1027; 2023-04-19 07:54:22,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-19 07:54:22,489 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=109, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/8b5c45880d9346ce9967079f46cf6537 2023-04-19 07:54:22,511 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/22fdc7d5aeed4b4d83fa41a4aba4d36b 2023-04-19 07:54:22,534 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/ed3c4bddc09a40bda6b7cf57706f3410 2023-04-19 07:54:22,540 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/8b5c45880d9346ce9967079f46cf6537 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/8b5c45880d9346ce9967079f46cf6537 2023-04-19 07:54:22,545 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/8b5c45880d9346ce9967079f46cf6537, entries=150, sequenceid=109, filesize=25.1 K 2023-04-19 07:54:22,546 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/22fdc7d5aeed4b4d83fa41a4aba4d36b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/22fdc7d5aeed4b4d83fa41a4aba4d36b 2023-04-19 07:54:22,550 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/22fdc7d5aeed4b4d83fa41a4aba4d36b, entries=150, sequenceid=109, filesize=11.6 K 2023-04-19 07:54:22,551 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/ed3c4bddc09a40bda6b7cf57706f3410 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/ed3c4bddc09a40bda6b7cf57706f3410 2023-04-19 07:54:22,555 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/ed3c4bddc09a40bda6b7cf57706f3410, entries=150, sequenceid=109, filesize=11.6 K 2023-04-19 07:54:22,556 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=120.76 KB/123660 for 2117921701b6b45f9e01175de52f16a3 in 504ms, sequenceid=109, compaction requested=false 2023-04-19 07:54:22,556 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:22,557 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:22,557 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:22,557 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:22,557 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,557 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:22,557 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:22,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,558 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:22,559 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,559 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:22,561 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=127.47 KB heapSize=358.69 KB 2023-04-19 07:54:22,561 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:22,561 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:22,561 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:22,561 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:22,562 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:22,562 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:22,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:22,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ca47fb[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:22,562 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:22,562 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,563 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,564 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:22,564 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:22,564 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:22,564 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,564 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:22,567 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:22,567 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:22,567 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:22,568 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:22,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:22,572 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:22,572 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:22,572 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:22,572 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:22,572 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:22,575 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,575 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:22,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:22,575 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:22,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:22,577 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:22,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,579 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,582 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:22,582 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:22,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:22,582 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,582 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,582 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,582 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:22,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:22,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:22,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:22,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:22,583 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:22,583 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:22,589 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419d3022f449faa4f2eac22f813d70913de to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d3022f449faa4f2eac22f813d70913de 2023-04-19 07:54:22,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:22,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54052 deadline: 1681890922603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,683 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:22,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:22,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:22,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:22,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:22,687 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:22,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:22,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:22,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:22,688 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,688 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:22,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:22,689 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:22,689 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,689 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:22,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:22,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:22,690 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:22,691 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:22,691 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:22,691 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:22,691 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:22,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:22,693 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:22,694 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,694 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55cd44dc[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:22,695 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:22,695 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,696 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,696 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:22,696 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:22,696 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:22,697 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,697 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:22,698 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:22,698 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:22,698 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,698 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:22,700 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:22,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:22,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:22,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:22,700 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:22,701 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,702 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:22,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,702 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:22,702 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,703 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,707 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,707 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:22,707 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,707 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:22,707 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:22,707 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:22,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:22,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:22,707 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,708 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:22,708 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,708 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,708 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:22,708 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:22,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:22,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:22,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54052 deadline: 1681890922705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,808 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:22,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:22,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:22,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:22,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:22,812 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:22,812 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:22,812 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:22,812 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:22,812 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,812 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,813 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:22,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:22,814 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,814 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:22,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:22,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:22,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:22,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:22,815 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:22,815 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:22,815 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:22,815 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:22,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:22,816 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:22,816 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:22,816 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:22,816 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:22,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:22,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:22,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:22,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:22,817 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,817 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:22,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:22,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:22,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e501035[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:22,820 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:22,820 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,821 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,821 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:22,821 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:22,821 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:22,821 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,821 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:22,826 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:22,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:22,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:22,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:22,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:22,829 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:22,829 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:22,829 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:22,829 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:22,829 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:22,831 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,831 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:22,831 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:22,832 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,834 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,836 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:22,836 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:22,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:22,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:22,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:22,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:22,837 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:22,837 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:22,837 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:22,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:22,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:22,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54052 deadline: 1681890922910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,937 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:22,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:22,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:22,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:22,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:22,940 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:22,940 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:22,940 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:22,940 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:22,941 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,941 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,942 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:22,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:22,942 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,942 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:22,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:22,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:22,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:22,944 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:22,944 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:22,944 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:22,944 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:22,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:22,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,947 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:22,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:22,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:22,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:22,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a5044fd[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:22,948 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:22,949 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,949 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,950 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:22,950 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:22,950 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:22,950 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,950 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:22,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:22,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:22,951 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:22,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:22,954 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:22,954 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:22,954 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:22,954 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:22,954 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:22,955 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,955 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:22,955 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:22,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:22,956 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,960 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,960 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:22,960 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,960 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:22,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:22,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:22,960 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:22,960 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:22,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:22,960 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:22,960 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,960 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:22,960 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:22,960 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:22,960 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:22,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:23,003 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=179, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/af811d918a734f06876708092d57fccb 2023-04-19 07:54:23,061 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:23,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:23,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:23,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:23,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:23,064 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:23,064 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:23,064 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:23,064 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:23,065 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,065 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,066 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,066 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:23,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:23,066 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,066 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:23,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:23,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:23,067 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:23,067 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:23,067 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:23,068 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:23,068 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:23,068 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:23,068 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:23,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:23,070 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,070 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:23,070 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,070 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:23,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:23,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:23,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d722fd3[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:23,071 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:23,072 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,073 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,073 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,073 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:23,073 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:23,073 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:23,073 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:23,076 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:23,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:23,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:23,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:23,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:23,079 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:23,079 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:23,079 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:23,079 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:23,079 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:23,080 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,080 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,080 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:23,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:23,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:23,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:23,082 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:23,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:23,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,086 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:23,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:23,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:23,086 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:23,086 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:23,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:23,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:23,186 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:23,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:23,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:23,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:23,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:23,190 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:23,190 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:23,190 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:23,190 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:23,191 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,191 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,192 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,192 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:23,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:23,192 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,192 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:23,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:23,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:23,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:23,194 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:23,194 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:23,194 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:23,194 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:23,196 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:23,197 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,197 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62a67b05[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:23,198 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:23,198 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,199 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:23,200 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,200 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:23,200 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:23,201 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,201 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:23,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:23,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:23,201 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:23,201 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:23,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:23,204 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:23,204 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:23,204 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:23,204 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:23,204 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:23,208 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:23,208 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,208 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:23,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:23,209 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,210 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,212 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:23,212 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:23,212 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:23,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:23,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:23,213 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:23,212 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,213 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,213 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,213 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,213 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,213 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,213 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:23,213 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:23,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:23,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:23,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:23,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54052 deadline: 1681890923216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,313 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:23,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:23,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:23,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:23,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:23,319 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:23,319 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:23,319 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:23,319 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:23,319 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,319 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:23,321 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:23,321 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,321 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:23,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:23,322 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:23,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:23,322 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:23,322 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:23,322 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:23,323 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:23,323 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:23,323 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:23,323 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:23,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:23,324 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,324 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:23,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:23,325 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:23,325 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:23,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:23,326 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:23,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd90edc[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:23,326 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,328 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:23,328 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,328 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:23,330 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:23,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:23,332 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:23,332 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:23,332 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:23,332 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:23,333 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:23,334 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:23,334 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,334 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:23,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:23,335 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,336 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,338 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,338 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,338 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,338 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,338 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,338 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:23,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:23,338 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:23,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:23,338 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,338 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:23,339 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:23,339 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:23,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:23,339 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:23,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:23,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/f0f568b5a367498b88991e25c16cb58d 2023-04-19 07:54:23,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/e989072bca4c4282868456b81b544073 2023-04-19 07:54:23,439 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:23,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:23,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:23,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/af811d918a734f06876708092d57fccb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/af811d918a734f06876708092d57fccb 2023-04-19 07:54:23,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:23,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:23,448 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:23,448 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:23,448 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:23,448 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:23,449 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,449 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,451 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:23,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:23,452 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,453 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,453 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:23,453 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/af811d918a734f06876708092d57fccb, entries=200, sequenceid=179, filesize=32.1 K 2023-04-19 07:54:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:23,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/f0f568b5a367498b88991e25c16cb58d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/f0f568b5a367498b88991e25c16cb58d 2023-04-19 07:54:23,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:23,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:23,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:23,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:23,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:23,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:23,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:23,457 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:23,457 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:23,457 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:23,457 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:23,458 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:23,459 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:23,460 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,460 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:23,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:23,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4426dc16[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:23,462 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:23,462 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,463 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:23,463 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/f0f568b5a367498b88991e25c16cb58d, entries=150, sequenceid=179, filesize=11.8 K 2023-04-19 07:54:23,465 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,465 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:23,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/e989072bca4c4282868456b81b544073 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/e989072bca4c4282868456b81b544073 2023-04-19 07:54:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:23,465 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:23,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:23,469 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:23,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:23,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:23,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:23,469 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:23,473 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,473 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,473 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:23,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:23,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:23,474 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,475 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,478 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:23,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:23,478 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:23,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:23,478 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:23,478 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:23,478 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,478 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,478 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,478 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:23,479 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:23,478 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,479 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:23,479 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:23,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:23,481 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/e989072bca4c4282868456b81b544073, entries=150, sequenceid=179, filesize=11.8 K 2023-04-19 07:54:23,482 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~358.64 KB/367248, currentSize=53.67 KB/54960 for 2117921701b6b45f9e01175de52f16a3 in 920ms, sequenceid=179, compaction requested=true 2023-04-19 07:54:23,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:23,482 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:23,482 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:54:23,482 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:54:23,482 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:23,482 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:23,484 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84516 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:54:23,484 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/A is initiating minor compaction (all files) 2023-04-19 07:54:23,484 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/A in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:23,484 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a36eace486cb44a8a90f6af9050f5b4d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/8b5c45880d9346ce9967079f46cf6537, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/af811d918a734f06876708092d57fccb] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=82.5 K 2023-04-19 07:54:23,484 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:54:23,484 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/B is initiating minor compaction (all files) 2023-04-19 07:54:23,484 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/B in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:23,484 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2c88b8d835f44af882a160638ab50d99, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/22fdc7d5aeed4b4d83fa41a4aba4d36b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/f0f568b5a367498b88991e25c16cb58d] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=35.1 K 2023-04-19 07:54:23,485 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 2c88b8d835f44af882a160638ab50d99, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681890860945 2023-04-19 07:54:23,485 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting a36eace486cb44a8a90f6af9050f5b4d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681890860945 2023-04-19 07:54:23,486 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 22fdc7d5aeed4b4d83fa41a4aba4d36b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681890861063 2023-04-19 07:54:23,486 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 8b5c45880d9346ce9967079f46cf6537, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681890861063 2023-04-19 07:54:23,486 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting af811d918a734f06876708092d57fccb, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681890862528 2023-04-19 07:54:23,486 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting f0f568b5a367498b88991e25c16cb58d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681890862528 2023-04-19 07:54:23,501 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#B#compaction#133 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:23,518 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#A#compaction#132 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:23,579 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:23,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:23,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:23,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:23,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:23,582 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:23,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:23,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:23,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:23,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,583 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,584 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,584 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:23,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:23,584 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,584 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:23,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:23,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:23,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:23,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:23,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:23,585 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:23,586 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:23,586 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:23,586 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:23,586 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:23,586 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 07:54:23,586 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:23,586 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:23,586 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:23,586 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:23,586 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:23,586 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:23,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:23,722 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:23,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890863741, max timestamp=1681890863765Num uniques 859; 2023-04-19 07:54:23,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890863741, max timestamp=1681890863765Num uniques 859; 2023-04-19 07:54:23,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890863741, max timestamp=1681890863765Num uniques 859; 2023-04-19 07:54:23,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:54:23,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:54:23,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:54:23,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:23,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54052 deadline: 1681890923816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,922 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/aa88b9a4ce9d4a0e8ca246b93b7a2108 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/aa88b9a4ce9d4a0e8ca246b93b7a2108 2023-04-19 07:54:23,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:23,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54052 deadline: 1681890923922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:23,927 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/B of 2117921701b6b45f9e01175de52f16a3 into aa88b9a4ce9d4a0e8ca246b93b7a2108(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:23,927 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:23,927 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/B, priority=13, startTime=1681890863482; duration=0sec 2023-04-19 07:54:23,928 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:23,928 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:54:23,929 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:54:23,929 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/C is initiating minor compaction (all files) 2023-04-19 07:54:23,929 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/C in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:23,929 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/de181201bf6e435689c0335a2290565b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/ed3c4bddc09a40bda6b7cf57706f3410, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/e989072bca4c4282868456b81b544073] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=35.1 K 2023-04-19 07:54:23,930 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting de181201bf6e435689c0335a2290565b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681890860945 2023-04-19 07:54:23,930 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting ed3c4bddc09a40bda6b7cf57706f3410, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681890861063 2023-04-19 07:54:23,930 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting e989072bca4c4282868456b81b544073, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681890862528 2023-04-19 07:54:23,937 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#C#compaction#135 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:23,957 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/b5bc9937e978476eb0b7c9d32d2557cb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b5bc9937e978476eb0b7c9d32d2557cb 2023-04-19 07:54:23,962 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/C of 2117921701b6b45f9e01175de52f16a3 into b5bc9937e978476eb0b7c9d32d2557cb(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:23,963 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:23,963 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/C, priority=13, startTime=1681890863482; duration=0sec 2023-04-19 07:54:23,963 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/5c77c9a0cb2e4081a4fdd2e7d4729551 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/5c77c9a0cb2e4081a4fdd2e7d4729551 2023-04-19 07:54:23,963 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:23,968 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/A of 2117921701b6b45f9e01175de52f16a3 into 5c77c9a0cb2e4081a4fdd2e7d4729551(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:23,968 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:23,968 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/A, priority=13, startTime=1681890863482; duration=0sec 2023-04-19 07:54:23,968 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:24,007 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419b1996960feaa487f9ff1cedf33b50448 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b1996960feaa487f9ff1cedf33b50448 2023-04-19 07:54:24,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:24,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54052 deadline: 1681890924125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:24,413 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=190, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/6527752509744515a433a57f89a81c73 2023-04-19 07:54:24,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:24,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54052 deadline: 1681890924426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:24,825 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/769e3ea7607c482fabd8174336083cb3 2023-04-19 07:54:24,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:24,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54052 deadline: 1681890924929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,237 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/830cf20241e14bbd87de97101b38829b 2023-04-19 07:54:25,242 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/6527752509744515a433a57f89a81c73 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/6527752509744515a433a57f89a81c73 2023-04-19 07:54:25,247 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/6527752509744515a433a57f89a81c73, entries=150, sequenceid=190, filesize=25.3 K 2023-04-19 07:54:25,247 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/769e3ea7607c482fabd8174336083cb3 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/769e3ea7607c482fabd8174336083cb3 2023-04-19 07:54:25,252 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/769e3ea7607c482fabd8174336083cb3, entries=150, sequenceid=190, filesize=11.8 K 2023-04-19 07:54:25,253 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/830cf20241e14bbd87de97101b38829b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/830cf20241e14bbd87de97101b38829b 2023-04-19 07:54:25,257 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/830cf20241e14bbd87de97101b38829b, entries=150, sequenceid=190, filesize=11.8 K 2023-04-19 07:54:25,258 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2117921701b6b45f9e01175de52f16a3 in 1672ms, sequenceid=190, compaction requested=false 2023-04-19 07:54:25,258 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:25,258 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:25,258 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:25,258 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:25,258 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,260 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:25,260 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,261 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:25,262 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,262 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:25,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:25,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:25,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8667ff6[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:25,263 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:25,263 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,266 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,266 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:25,266 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,266 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:25,266 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:25,266 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:25,268 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:25,268 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:25,268 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,268 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:25,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:25,270 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:25,270 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:25,270 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:25,270 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:25,270 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:25,272 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,272 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:25,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,272 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:25,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:25,273 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,274 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:25,285 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,285 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:25,285 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:25,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:25,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:25,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:25,285 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:25,285 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:25,285 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,285 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,285 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,285 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:25,286 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:25,285 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:25,386 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:25,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:25,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:25,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:25,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:25,389 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:25,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:25,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:25,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:25,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,390 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:25,391 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,391 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:25,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:25,391 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,391 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:25,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:25,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:25,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:25,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:25,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:25,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:25,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:25,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:25,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:25,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:25,393 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:25,393 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:25,393 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-19 07:54:25,394 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:25,394 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:25,394 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:25,394 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:25,394 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:25,394 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:25,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:25,764 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:25,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:25,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54044 deadline: 1681890925779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,783 DEBUG [Thread-863] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:25,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:25,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890925781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:25,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890925783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:25,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54054 deadline: 1681890925791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,793 DEBUG [Thread-859] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:25,810 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419248cfddefbaa4c8a81f09f73b129e871 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419248cfddefbaa4c8a81f09f73b129e871 2023-04-19 07:54:25,815 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=227, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/a2db6155b7e94aec8c0a3e0a0ad4785a 2023-04-19 07:54:25,845 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/eb33ebd2272747db8f8c95a362c4fc6b 2023-04-19 07:54:25,858 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/8e27891d701e4ed3b9507eb5ee8303fa 2023-04-19 07:54:25,862 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/a2db6155b7e94aec8c0a3e0a0ad4785a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a2db6155b7e94aec8c0a3e0a0ad4785a 2023-04-19 07:54:25,868 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a2db6155b7e94aec8c0a3e0a0ad4785a, entries=150, sequenceid=227, filesize=25.3 K 2023-04-19 07:54:25,869 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/eb33ebd2272747db8f8c95a362c4fc6b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/eb33ebd2272747db8f8c95a362c4fc6b 2023-04-19 07:54:25,875 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/eb33ebd2272747db8f8c95a362c4fc6b, entries=150, sequenceid=227, filesize=11.8 K 2023-04-19 07:54:25,876 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/8e27891d701e4ed3b9507eb5ee8303fa as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/8e27891d701e4ed3b9507eb5ee8303fa 2023-04-19 07:54:25,884 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/8e27891d701e4ed3b9507eb5ee8303fa, entries=150, sequenceid=227, filesize=11.8 K 2023-04-19 07:54:25,885 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=60.38 KB/61830 for 2117921701b6b45f9e01175de52f16a3 in 492ms, sequenceid=227, compaction requested=true 2023-04-19 07:54:25,885 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:25,885 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:25,885 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:25,885 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:25,885 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,885 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:25,885 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:25,887 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,887 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:25,887 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,887 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:25,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:25,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59c943ac[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:25,889 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:25,889 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,890 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,891 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-19 07:54:25,891 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,891 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:25,891 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:25,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:25,891 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:25,891 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:25,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:25,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:25,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:25,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:25,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:25,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:25,894 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:25,894 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:25,894 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:25,894 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:25,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:25,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:25,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:25,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:25,899 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:25,899 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:25,899 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:25,899 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:25,899 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:25,901 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,901 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:25,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,901 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:25,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,903 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:25,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,904 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,907 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:25,907 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:25,907 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,907 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:25,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:25,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:25,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:25,907 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:25,908 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:25,907 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:25,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:25,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:25,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:25,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:25,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:25,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890925918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:25,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890925921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:25,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:25,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54052 deadline: 1681890925936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,008 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:26,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:26,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:26,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:26,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:26,011 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:26,011 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:26,011 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:26,011 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:26,011 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,011 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,012 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:26,012 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:26,013 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,013 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:26,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:26,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:26,014 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:26,014 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:26,014 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:26,015 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:26,015 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:26,015 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:26,015 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:26,015 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:26,015 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:26,015 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:26,015 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:26,015 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,015 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:26,015 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:26,017 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,018 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,018 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,018 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5de209a7[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:26,020 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:26,020 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,021 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:26,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:26,021 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,021 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:26,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:26,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:26,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:26,023 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890926022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:26,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:26,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:26,026 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:26,026 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:26,026 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:26,026 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:26,026 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:26,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:26,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890926024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,027 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,027 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,027 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:26,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:26,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,028 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:26,029 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:26,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:26,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:26,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:26,033 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:26,033 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:26,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:26,133 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:26,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:26,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:26,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:26,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:26,136 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:26,136 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:26,136 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:26,136 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:26,136 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,136 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,138 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:26,138 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:26,138 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,138 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:26,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:26,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:26,140 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:26,140 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:26,140 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:26,140 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:26,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:26,142 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,142 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:26,142 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,142 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:26,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:26,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fd6b861[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:26,144 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:26,144 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,145 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:26,145 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,145 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:26,147 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:26,147 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,147 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,147 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:26,151 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:26,151 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:26,151 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:26,151 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:26,151 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:26,152 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,152 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,152 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:26,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:26,153 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,154 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:26,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,160 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,160 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,160 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,160 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,160 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:26,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:26,160 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:26,160 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:26,160 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:26,161 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,161 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:26,161 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:26,161 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:26,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:26,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:26,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890926225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:26,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890926228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,261 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:26,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:26,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:26,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:26,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:26,264 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:26,264 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:26,264 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:26,264 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:26,265 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,265 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,267 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,267 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:26,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:26,267 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,267 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:26,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:26,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:26,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:26,268 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:26,268 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:26,269 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:26,269 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:26,269 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:26,269 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:26,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,270 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:26,270 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,270 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:26,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a5180c3[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:26,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:26,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,273 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:26,273 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,273 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:26,273 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:26,274 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:26,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:26,274 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:26,275 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:26,275 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:26,277 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:26,277 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:26,277 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:26,277 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:26,277 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:26,279 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,279 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,279 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:26,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:26,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:26,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:26,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:26,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:26,280 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,281 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:26,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:26,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:26,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:26,284 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:26,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:26,284 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:26,284 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:26,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:26,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:26,323 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419614434a8fb6944658e413b29d9eb5199 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419614434a8fb6944658e413b29d9eb5199 2023-04-19 07:54:26,329 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=241, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/0f4fad54502b450c9c392a410039b502 2023-04-19 07:54:26,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/b28f11b230d84ef5b60fa9f8e0d1349c 2023-04-19 07:54:26,353 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/22bc4040d80d4544b286524aff2d8253 2023-04-19 07:54:26,358 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/0f4fad54502b450c9c392a410039b502 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/0f4fad54502b450c9c392a410039b502 2023-04-19 07:54:26,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/0f4fad54502b450c9c392a410039b502, entries=250, sequenceid=241, filesize=38.9 K 2023-04-19 07:54:26,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/b28f11b230d84ef5b60fa9f8e0d1349c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/b28f11b230d84ef5b60fa9f8e0d1349c 2023-04-19 07:54:26,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/b28f11b230d84ef5b60fa9f8e0d1349c, entries=150, sequenceid=241, filesize=11.8 K 2023-04-19 07:54:26,368 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/22bc4040d80d4544b286524aff2d8253 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/22bc4040d80d4544b286524aff2d8253 2023-04-19 07:54:26,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/22bc4040d80d4544b286524aff2d8253, entries=150, sequenceid=241, filesize=11.8 K 2023-04-19 07:54:26,379 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 2117921701b6b45f9e01175de52f16a3 in 489ms, sequenceid=241, compaction requested=true 2023-04-19 07:54:26,379 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:26,379 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:26,379 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:26,379 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:26,379 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:26,379 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:26,381 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48384 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:26,381 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/B is initiating minor compaction (all files) 2023-04-19 07:54:26,381 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/B in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:26,382 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/aa88b9a4ce9d4a0e8ca246b93b7a2108, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/769e3ea7607c482fabd8174336083cb3, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/eb33ebd2272747db8f8c95a362c4fc6b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/b28f11b230d84ef5b60fa9f8e0d1349c] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=47.3 K 2023-04-19 07:54:26,382 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 117808 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:26,382 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/A is initiating minor compaction (all files) 2023-04-19 07:54:26,382 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/A in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:26,382 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/5c77c9a0cb2e4081a4fdd2e7d4729551, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/6527752509744515a433a57f89a81c73, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a2db6155b7e94aec8c0a3e0a0ad4785a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/0f4fad54502b450c9c392a410039b502] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=115.0 K 2023-04-19 07:54:26,383 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting aa88b9a4ce9d4a0e8ca246b93b7a2108, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681890862528 2023-04-19 07:54:26,383 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 5c77c9a0cb2e4081a4fdd2e7d4729551, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681890862528 2023-04-19 07:54:26,383 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 6527752509744515a433a57f89a81c73, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1681890862586 2023-04-19 07:54:26,383 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 769e3ea7607c482fabd8174336083cb3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1681890862586 2023-04-19 07:54:26,383 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting a2db6155b7e94aec8c0a3e0a0ad4785a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1681890863809 2023-04-19 07:54:26,384 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting eb33ebd2272747db8f8c95a362c4fc6b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1681890863809 2023-04-19 07:54:26,384 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 0f4fad54502b450c9c392a410039b502, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1681890865771 2023-04-19 07:54:26,385 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:26,385 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting b28f11b230d84ef5b60fa9f8e0d1349c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1681890865771 2023-04-19 07:54:26,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:26,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:26,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:26,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:26,388 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:26,388 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:26,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:26,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:26,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,391 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,391 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:26,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:26,391 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,391 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:26,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:26,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:26,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:26,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:26,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:26,396 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:26,396 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:26,396 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:26,396 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:26,396 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-19 07:54:26,396 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:26,397 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:26,397 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:26,397 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:26,397 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:26,397 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:26,405 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#B#compaction#144 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:26,440 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#A#compaction#145 average throughput is 0.58 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:26,470 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419f84bd1c69a014a10a305c7c1c42faf5b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f84bd1c69a014a10a305c7c1c42faf5b 2023-04-19 07:54:26,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:26,530 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:26,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:26,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890926549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:26,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890926549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:26,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890926650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:26,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890926652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:26,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890926853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,858 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/8b8b9bcd7f62406abf559efea7d15a79 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/8b8b9bcd7f62406abf559efea7d15a79 2023-04-19 07:54:26,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:26,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890926860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,863 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/B of 2117921701b6b45f9e01175de52f16a3 into 8b8b9bcd7f62406abf559efea7d15a79(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:26,864 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:26,864 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/B, priority=12, startTime=1681890866379; duration=0sec 2023-04-19 07:54:26,864 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:26,864 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:26,865 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48384 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:26,866 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/C is initiating minor compaction (all files) 2023-04-19 07:54:26,866 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/C in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:26,866 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b5bc9937e978476eb0b7c9d32d2557cb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/830cf20241e14bbd87de97101b38829b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/8e27891d701e4ed3b9507eb5ee8303fa, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/22bc4040d80d4544b286524aff2d8253] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=47.3 K 2023-04-19 07:54:26,866 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting b5bc9937e978476eb0b7c9d32d2557cb, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1681890862528 2023-04-19 07:54:26,867 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 830cf20241e14bbd87de97101b38829b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1681890862586 2023-04-19 07:54:26,867 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 8e27891d701e4ed3b9507eb5ee8303fa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1681890863809 2023-04-19 07:54:26,868 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 22bc4040d80d4544b286524aff2d8253, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1681890865771 2023-04-19 07:54:26,874 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#C#compaction#147 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:26,883 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/aaa9fe070ac543e38edf46e459ab950d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/aaa9fe070ac543e38edf46e459ab950d 2023-04-19 07:54:26,889 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/C of 2117921701b6b45f9e01175de52f16a3 into aaa9fe070ac543e38edf46e459ab950d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:26,889 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:26,889 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/C, priority=12, startTime=1681890866379; duration=0sec 2023-04-19 07:54:26,889 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:26,890 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=259, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/3ea3d93837894ff5a79bf264253e306b 2023-04-19 07:54:26,910 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/ebe0e8cda53f4904a9c436f8145f73b6 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/ebe0e8cda53f4904a9c436f8145f73b6 2023-04-19 07:54:26,911 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/448f25cee33a405ca9c8dea97557179e 2023-04-19 07:54:26,918 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/A of 2117921701b6b45f9e01175de52f16a3 into ebe0e8cda53f4904a9c436f8145f73b6(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:26,918 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:26,918 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/A, priority=12, startTime=1681890866379; duration=0sec 2023-04-19 07:54:26,918 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:26,939 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/75232f2a29ae423b83fdfcfcd3bec030 2023-04-19 07:54:26,944 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/3ea3d93837894ff5a79bf264253e306b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/3ea3d93837894ff5a79bf264253e306b 2023-04-19 07:54:26,948 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/3ea3d93837894ff5a79bf264253e306b, entries=150, sequenceid=259, filesize=25.3 K 2023-04-19 07:54:26,949 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/448f25cee33a405ca9c8dea97557179e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/448f25cee33a405ca9c8dea97557179e 2023-04-19 07:54:26,954 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/448f25cee33a405ca9c8dea97557179e, entries=150, sequenceid=259, filesize=11.8 K 2023-04-19 07:54:26,956 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/75232f2a29ae423b83fdfcfcd3bec030 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/75232f2a29ae423b83fdfcfcd3bec030 2023-04-19 07:54:26,962 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/75232f2a29ae423b83fdfcfcd3bec030, entries=150, sequenceid=259, filesize=11.8 K 2023-04-19 07:54:26,963 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 2117921701b6b45f9e01175de52f16a3 in 567ms, sequenceid=259, compaction requested=false 2023-04-19 07:54:26,963 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:26,963 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:26,963 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:26,964 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:26,964 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,964 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:26,964 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:26,965 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,965 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:26,966 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,966 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:26,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:26,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50de2c58[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:26,968 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:26,968 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,971 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:26,971 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,971 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:26,972 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:26,973 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:26,973 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:26,973 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:26,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:26,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:26,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:26,976 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:26,976 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:26,976 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:26,976 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:26,976 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:26,978 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,978 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,978 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:26,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:26,979 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:26,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,980 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:26,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:26,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:26,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:26,984 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:26,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:26,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:26,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:26,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:26,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:26,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:26,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:26,985 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:26,985 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:27,085 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:27,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:27,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:27,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:27,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:27,089 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:27,089 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:27,089 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:27,089 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:27,089 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:27,090 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:27,092 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:27,092 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:27,093 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,093 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:27,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:27,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:27,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:27,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:27,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:27,094 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:27,094 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:27,094 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:27,100 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:27,100 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:27,100 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:27,100 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:27,100 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-19 07:54:27,100 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:27,100 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:27,101 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:27,101 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:27,101 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:27,101 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:27,118 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041946f2cad00f8a4bc89a1979a9ed6a497b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041946f2cad00f8a4bc89a1979a9ed6a497b 2023-04-19 07:54:27,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:27,166 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:27,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:27,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890927183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:27,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890927185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:27,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890927286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:27,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890927286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:27,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890927489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:27,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890927489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,525 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=276, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/657e18982de34d869475d26e1a72d8d9 2023-04-19 07:54:27,538 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/1586696512b74bcfa1b4294f2fb98339 2023-04-19 07:54:27,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:27,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890927793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:27,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890927794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:27,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54052 deadline: 1681890927945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,948 DEBUG [Thread-867] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4131 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:27,950 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/8664f4bcc2fc486ebe57d18b79cdd4bf 2023-04-19 07:54:27,954 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/657e18982de34d869475d26e1a72d8d9 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/657e18982de34d869475d26e1a72d8d9 2023-04-19 07:54:27,959 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/657e18982de34d869475d26e1a72d8d9, entries=150, sequenceid=276, filesize=25.4 K 2023-04-19 07:54:27,959 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/1586696512b74bcfa1b4294f2fb98339 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/1586696512b74bcfa1b4294f2fb98339 2023-04-19 07:54:27,964 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/1586696512b74bcfa1b4294f2fb98339, entries=150, sequenceid=276, filesize=11.9 K 2023-04-19 07:54:27,965 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/8664f4bcc2fc486ebe57d18b79cdd4bf as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/8664f4bcc2fc486ebe57d18b79cdd4bf 2023-04-19 07:54:27,970 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/8664f4bcc2fc486ebe57d18b79cdd4bf, entries=150, sequenceid=276, filesize=11.9 K 2023-04-19 07:54:27,971 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 2117921701b6b45f9e01175de52f16a3 in 871ms, sequenceid=276, compaction requested=true 2023-04-19 07:54:27,971 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:27,971 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:27,971 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:27,971 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:27,971 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,971 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:27,971 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:27,973 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:27,973 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:27,973 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:27,973 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:27,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:27,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:27,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6344641a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:27,975 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:27,975 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:27,976 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:27,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:27,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:27,976 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:27,976 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:27,976 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:27,976 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,976 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:27,977 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:27,977 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:27,977 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,977 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:27,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:27,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:27,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:27,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:27,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:27,980 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:27,980 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:27,980 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:27,980 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:27,980 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:27,982 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:27,982 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:27,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:27,982 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:27,983 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,984 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:27,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:27,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,986 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:27,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:27,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:27,987 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:27,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:27,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:27,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:27,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:27,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:27,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:27,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:27,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:27,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:27,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:27,987 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:27,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:27,987 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:28,087 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:28,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:28,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:28,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:28,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:28,091 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:28,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:28,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:28,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:28,092 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:28,092 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:28,093 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:28,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:28,093 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:28,093 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,094 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:28,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:28,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:28,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:28,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:28,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:28,095 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:28,095 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:28,095 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:28,095 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:28,095 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:28,095 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:28,095 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:28,095 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-19 07:54:28,095 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:28,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:28,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:28,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:28,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:28,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:28,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:28,298 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:28,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:28,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890928342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:28,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890928342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:28,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890928447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:28,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890928448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,511 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304191c880dd7a6164ccc868f39bb157ea734 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191c880dd7a6164ccc868f39bb157ea734 2023-04-19 07:54:28,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:28,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890928651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:28,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890928651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,917 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=294, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/a4bf4faef7bb41c1bb81ea418c1529cc 2023-04-19 07:54:28,929 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/c62b7b8dd0d943bb986566917235bab1 2023-04-19 07:54:28,940 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/10cb02bc1f7748a693fe505749420dc7 2023-04-19 07:54:28,944 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/a4bf4faef7bb41c1bb81ea418c1529cc as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a4bf4faef7bb41c1bb81ea418c1529cc 2023-04-19 07:54:28,949 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a4bf4faef7bb41c1bb81ea418c1529cc, entries=150, sequenceid=294, filesize=25.4 K 2023-04-19 07:54:28,950 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/c62b7b8dd0d943bb986566917235bab1 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c62b7b8dd0d943bb986566917235bab1 2023-04-19 07:54:28,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:28,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890928953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,955 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c62b7b8dd0d943bb986566917235bab1, entries=150, sequenceid=294, filesize=11.9 K 2023-04-19 07:54:28,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:28,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890928953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,955 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/10cb02bc1f7748a693fe505749420dc7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/10cb02bc1f7748a693fe505749420dc7 2023-04-19 07:54:28,960 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/10cb02bc1f7748a693fe505749420dc7, entries=150, sequenceid=294, filesize=11.9 K 2023-04-19 07:54:28,960 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 2117921701b6b45f9e01175de52f16a3 in 865ms, sequenceid=294, compaction requested=true 2023-04-19 07:54:28,960 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:28,961 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:28,961 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:28,961 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:28,961 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,961 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:28,961 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:28,966 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,966 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:28,966 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:28,966 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:28,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:28,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c1da857[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:28,968 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:28,968 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:28,969 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:28,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:28,970 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:28,970 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:28,970 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:28,970 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,970 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:28,973 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,973 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:28,973 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:28,973 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:28,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:28,979 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:28,979 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:28,979 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:28,979 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:28,979 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:28,982 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:28,982 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:28,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:28,982 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:28,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:28,983 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,984 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:28,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:28,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:28,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:28,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:28,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:28,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:28,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:28,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:28,987 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:28,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:28,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:28,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:28,987 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:28,987 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:28,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:29,087 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:29,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:29,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:29,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:29,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:29,091 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:29,092 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:29,092 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:29,092 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:29,092 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:29,092 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:29,094 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:29,094 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:29,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:29,094 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:29,094 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:29,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:29,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:29,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:29,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:29,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:29,095 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:29,095 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:29,096 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:29,096 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:29,096 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:29,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:29,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:29,096 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-19 07:54:29,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:29,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:29,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:29,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:29,097 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:29,097 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:29,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:29,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:29,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:29,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890929484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:29,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:29,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890929487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:29,516 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419d1d0ffe5b2194564b573e2cf29ac02a3 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d1d0ffe5b2194564b573e2cf29ac02a3 2023-04-19 07:54:29,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:29,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890929591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:29,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:29,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890929593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:29,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:29,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890929793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:29,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:29,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890929800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:29,933 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=309, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/ee3c96969fcd4807bfee4b16c63e28a8 2023-04-19 07:54:30,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:30,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890930095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:30,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890930107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,347 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/b1e7f57489c34865ad4c32b543d4a5f9 2023-04-19 07:54:30,360 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/5659a9d3d01e4851b009d56d3895e072 2023-04-19 07:54:30,364 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/ee3c96969fcd4807bfee4b16c63e28a8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/ee3c96969fcd4807bfee4b16c63e28a8 2023-04-19 07:54:30,368 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/ee3c96969fcd4807bfee4b16c63e28a8, entries=150, sequenceid=309, filesize=25.4 K 2023-04-19 07:54:30,369 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/b1e7f57489c34865ad4c32b543d4a5f9 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/b1e7f57489c34865ad4c32b543d4a5f9 2023-04-19 07:54:30,374 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/b1e7f57489c34865ad4c32b543d4a5f9, entries=150, sequenceid=309, filesize=11.9 K 2023-04-19 07:54:30,375 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/5659a9d3d01e4851b009d56d3895e072 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/5659a9d3d01e4851b009d56d3895e072 2023-04-19 07:54:30,380 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/5659a9d3d01e4851b009d56d3895e072, entries=150, sequenceid=309, filesize=11.9 K 2023-04-19 07:54:30,382 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2117921701b6b45f9e01175de52f16a3 in 1286ms, sequenceid=309, compaction requested=true 2023-04-19 07:54:30,382 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:30,382 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:30,382 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:30,384 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:30,384 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:30,384 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:30,384 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:30,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:30,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:30,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a8c788f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:30,386 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:30,386 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:30,388 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:30,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:30,388 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,388 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:30,390 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:30,390 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:30,390 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,390 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:30,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:30,392 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:30,393 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:30,393 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:30,393 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:30,393 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:30,394 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:30,394 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:30,394 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:30,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:30,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:30,395 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,397 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:30,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:30,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:30,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:30,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:30,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:30,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:30,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:30,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:30,399 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:30,399 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:30,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:30,400 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:30,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:30,500 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:30,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:30,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:30,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:30,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:30,503 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:30,503 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:30,503 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:30,504 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:30,504 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:30,504 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:30,506 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:30,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:30,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:30,507 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,507 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:30,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:30,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:30,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:30,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:30,508 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:30,508 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:30,508 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-19 07:54:30,509 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:30,509 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:30,509 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:30,509 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:30,509 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:30,509 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:30,529 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419475775ce722744e4932d0b09915c381a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419475775ce722744e4932d0b09915c381a 2023-04-19 07:54:30,535 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=326, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/c11ee6464acc442a8f28156bd3898c61 2023-04-19 07:54:30,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:30,602 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:30,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:30,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890930631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:30,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890930634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:30,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890930736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:30,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890930742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:30,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890930937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:30,947 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/c2be367a6fac41d09e134924e5221d46 2023-04-19 07:54:30,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:30,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890930947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:31,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54058 deadline: 1681890931241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:31,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890931255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,360 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/374ee43263ed419fa062d00053e9ce19 2023-04-19 07:54:31,365 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/c11ee6464acc442a8f28156bd3898c61 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/c11ee6464acc442a8f28156bd3898c61 2023-04-19 07:54:31,369 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/c11ee6464acc442a8f28156bd3898c61, entries=150, sequenceid=326, filesize=25.4 K 2023-04-19 07:54:31,370 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/c2be367a6fac41d09e134924e5221d46 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c2be367a6fac41d09e134924e5221d46 2023-04-19 07:54:31,375 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c2be367a6fac41d09e134924e5221d46, entries=150, sequenceid=326, filesize=11.9 K 2023-04-19 07:54:31,376 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/374ee43263ed419fa062d00053e9ce19 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/374ee43263ed419fa062d00053e9ce19 2023-04-19 07:54:31,382 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/374ee43263ed419fa062d00053e9ce19, entries=150, sequenceid=326, filesize=11.9 K 2023-04-19 07:54:31,382 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2117921701b6b45f9e01175de52f16a3 in 874ms, sequenceid=326, compaction requested=true 2023-04-19 07:54:31,383 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:31,383 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:31,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:31,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:31,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,384 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:31,384 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:31,386 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,386 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:31,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:31,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:31,386 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:31,386 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:31,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:31,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:31,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:31,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b112f6a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:31,388 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:31,388 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:31,390 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:31,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:31,390 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:31,390 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:31,390 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:31,390 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,390 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:31,392 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:31,392 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,392 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:31,392 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:31,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:31,394 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:31,394 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:31,394 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:31,394 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:31,394 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:31,397 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:31,397 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:31,397 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:31,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:31,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:31,398 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:31,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:31,399 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:31,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,401 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,401 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:31,401 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:31,401 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:31,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:31,401 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:31,401 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:31,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:31,401 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:31,401 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:31,401 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:31,402 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:31,401 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:31,402 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:31,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:31,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:31,502 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:31,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:31,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:31,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:31,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:31,505 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:31,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:31,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:31,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:31,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:31,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:31,507 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,507 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:31,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:31,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:31,508 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,508 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:31,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:31,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:31,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:31,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:31,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:31,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:31,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:31,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:31,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:31,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:31,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:31,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:31,509 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:31,509 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:31,509 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-19 07:54:31,510 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:31,510 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:31,510 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:31,510 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:31,510 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:31,510 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:31,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:31,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:31,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:31,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:31,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890931791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890931792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:31,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890931896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:31,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890931896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,926 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041922402447d06147aa82e0dee29347aa15 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041922402447d06147aa82e0dee29347aa15 2023-04-19 07:54:31,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:31,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54052 deadline: 1681890931987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:31,992 DEBUG [Thread-867] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:32,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:32,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890932100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:32,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:32,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890932100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:32,332 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=341, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/742a1f1e12eb4437831ff59350eec94e 2023-04-19 07:54:32,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:32,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890932405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:32,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:32,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890932405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:32,744 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/235cac3a50304e6f855c069f1d123c97 2023-04-19 07:54:32,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:32,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54018 deadline: 1681890932910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:32,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:32,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890932913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,155 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/3093e9ab35864ac5b744829848028a62 2023-04-19 07:54:33,161 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/742a1f1e12eb4437831ff59350eec94e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/742a1f1e12eb4437831ff59350eec94e 2023-04-19 07:54:33,168 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/742a1f1e12eb4437831ff59350eec94e, entries=150, sequenceid=341, filesize=25.4 K 2023-04-19 07:54:33,169 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/235cac3a50304e6f855c069f1d123c97 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/235cac3a50304e6f855c069f1d123c97 2023-04-19 07:54:33,173 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/235cac3a50304e6f855c069f1d123c97, entries=150, sequenceid=341, filesize=11.9 K 2023-04-19 07:54:33,174 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/3093e9ab35864ac5b744829848028a62 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/3093e9ab35864ac5b744829848028a62 2023-04-19 07:54:33,179 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/3093e9ab35864ac5b744829848028a62, entries=150, sequenceid=341, filesize=11.9 K 2023-04-19 07:54:33,179 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2117921701b6b45f9e01175de52f16a3 in 1670ms, sequenceid=341, compaction requested=true 2023-04-19 07:54:33,179 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:33,179 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:33,180 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:33,180 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:33,180 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,180 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:33,180 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:33,181 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,181 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:33,182 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:33,182 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:33,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:33,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:33,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bb5fca6[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:33,184 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:33,184 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:33,189 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:33,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:33,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:33,189 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:33,189 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:33,189 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:33,189 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,189 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:33,190 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:33,190 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:33,190 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,191 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:33,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:33,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:33,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:33,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:33,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:33,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:33,193 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:33,193 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:33,193 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:33,193 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:33,193 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:33,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:33,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:33,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:33,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:33,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:33,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:33,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:33,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:33,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:33,196 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:33,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:33,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,197 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:33,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:33,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,199 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,199 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:33,199 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:33,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:33,199 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:33,199 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:33,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:33,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:33,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:33,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:33,199 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:33,200 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,200 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:33,200 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:33,200 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:33,200 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:33,200 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:33,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:33,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:33,300 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:33,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:33,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:33,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:33,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:33,303 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:33,303 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:33,303 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:33,303 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:33,304 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:33,304 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:33,306 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,306 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:33,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:33,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:33,306 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,306 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:33,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:33,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:33,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:33,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:33,308 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:33,308 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:33,308 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:33,308 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:33,308 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:33,308 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:33,308 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:33,308 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-19 07:54:33,309 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:33,309 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:33,309 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:33,309 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:33,309 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:33,309 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:33,723 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304194749f96db1e847c0964a02ceba342b4e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194749f96db1e847c0964a02ceba342b4e 2023-04-19 07:54:33,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:33,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:33,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:33,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:33,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54018 deadline: 1681890933980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:33,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890933980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:34,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890934084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:34,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54018 deadline: 1681890934084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,129 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=358, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/fca0d09aac894790bc6060fa20edad6c 2023-04-19 07:54:34,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:34,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890934286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:34,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54018 deadline: 1681890934288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,541 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/6e977a4133a14e96b8ca32a6266895be 2023-04-19 07:54:34,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:34,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890934588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:34,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54018 deadline: 1681890934596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,958 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/87b67b8c7405443e923206b04cbc1f4f 2023-04-19 07:54:34,964 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/fca0d09aac894790bc6060fa20edad6c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/fca0d09aac894790bc6060fa20edad6c 2023-04-19 07:54:34,969 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/fca0d09aac894790bc6060fa20edad6c, entries=150, sequenceid=358, filesize=25.4 K 2023-04-19 07:54:34,970 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/6e977a4133a14e96b8ca32a6266895be as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/6e977a4133a14e96b8ca32a6266895be 2023-04-19 07:54:34,975 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/6e977a4133a14e96b8ca32a6266895be, entries=150, sequenceid=358, filesize=11.9 K 2023-04-19 07:54:34,976 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/87b67b8c7405443e923206b04cbc1f4f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/87b67b8c7405443e923206b04cbc1f4f 2023-04-19 07:54:34,981 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/87b67b8c7405443e923206b04cbc1f4f, entries=150, sequenceid=358, filesize=11.9 K 2023-04-19 07:54:34,981 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2117921701b6b45f9e01175de52f16a3 in 1673ms, sequenceid=358, compaction requested=true 2023-04-19 07:54:34,981 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:34,982 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:34,982 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:34,982 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:34,982 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,982 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:34,982 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:34,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,984 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:34,984 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:34,984 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:34,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:34,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4afba897[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:34,986 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:34,986 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:34,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:34,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:34,987 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:34,987 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:34,987 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:34,987 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,987 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:34,990 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,990 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:34,990 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:34,990 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:34,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:34,993 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:34,993 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:34,993 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:34,993 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:34,993 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:34,997 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:34,997 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:34,997 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:34,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:34,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:34,998 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,999 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:34,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,002 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,002 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,002 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,002 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,002 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,002 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:35,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:35,002 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,003 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,003 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,003 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,003 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:35,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,003 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:35,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:35,094 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 07:54:35,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:35,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:35,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:35,095 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:35,095 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:35,095 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:35,103 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:35,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:35,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:35,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:35,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:35,107 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:35,107 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:35,107 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:35,107 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:35,107 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,107 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,109 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,109 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,109 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:35,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:35,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:35,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:35,110 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:35,110 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:35,110 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:35,110 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:35,110 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:35,110 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:35,110 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:35,110 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:35,110 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:35,110 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:35,111 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:35,111 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,111 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:35,111 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:35,112 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,112 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,113 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,113 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:35,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13eaf947[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:35,114 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:35,114 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,116 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:35,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:35,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:35,117 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,117 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:35,118 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,118 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:35,118 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,118 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:35,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:35,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:35,121 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:35,121 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:35,121 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:35,121 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:35,121 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:35,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,124 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,125 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,127 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,128 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:35,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:35,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:35,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890935125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,128 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:35,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:35,128 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:35,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54018 deadline: 1681890935126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,129 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,229 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:35,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:35,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:35,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:35,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:35,232 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:35,232 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:35,232 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:35,232 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:35,233 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:35,233 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890935229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:35,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54018 deadline: 1681890935229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,235 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,235 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,235 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,235 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:35,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:35,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:35,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:35,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:35,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:35,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:35,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:35,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:35,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:35,237 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:35,237 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:35,237 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:35,237 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:35,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:35,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:35,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:35,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:35,239 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,239 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,239 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,239 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:35,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@112fe2f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:35,241 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:35,241 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,242 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:35,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:35,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:35,242 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,242 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:35,243 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,243 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:35,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:35,246 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:35,246 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:35,246 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:35,246 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:35,246 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:35,254 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,254 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,255 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,255 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,256 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,259 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,259 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:35,259 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,259 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,259 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,259 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:35,259 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:35,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,259 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,259 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:35,259 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:35,259 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,259 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,359 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:35,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:35,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:35,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:35,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:35,364 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:35,364 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:35,364 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:35,364 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:35,365 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,365 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,366 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,366 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,367 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,367 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:35,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:35,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:35,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:35,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:35,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:35,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:35,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:35,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:35,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:35,368 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:35,368 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:35,368 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:35,368 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:35,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:35,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:35,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:35,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:35,370 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,370 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,370 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,370 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:35,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75366267[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:35,372 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:35,372 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,374 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,374 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:35,374 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,374 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:35,374 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:35,374 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:35,376 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,376 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:35,376 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,376 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:35,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:35,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:35,379 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:35,379 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:35,379 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:35,380 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:35,380 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:35,382 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,382 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,382 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,382 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,384 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,387 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,387 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,387 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,387 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,387 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:35,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:35,387 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,387 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,387 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,387 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,387 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,387 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:35,388 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:35,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:35,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:35,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54018 deadline: 1681890935434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:35,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890935434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,488 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:35,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:35,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:35,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:35,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:35,491 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:35,491 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:35,491 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:35,491 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:35,492 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,492 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,495 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,495 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,495 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:35,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:35,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:35,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:35,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:35,496 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:35,496 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:35,496 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:35,497 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:35,497 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:35,497 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:35,497 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:35,497 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:35,497 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:35,497 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:35,497 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:35,497 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,497 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:35,497 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:35,498 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,499 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,499 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:35,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:35,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@629ec346[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:35,500 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:35,500 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,501 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,501 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:35,501 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:35,502 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:35,502 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,502 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:35,503 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:35,503 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,503 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,503 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:35,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:35,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:35,506 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:35,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:35,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:35,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:35,506 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:35,510 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,510 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,510 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,510 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,512 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,513 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304194fda67768778496da08c95fbc1dd89bf to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194fda67768778496da08c95fbc1dd89bf 2023-04-19 07:54:35,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,514 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,514 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:35,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:35,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,514 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,514 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,515 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:35,515 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:35,514 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,515 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,515 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,515 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,515 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,515 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:35,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,615 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:35,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:35,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:35,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:35,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:35,618 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:35,618 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:35,619 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:35,619 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:35,619 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,619 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,620 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,620 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,621 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,621 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:35,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:35,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:35,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:35,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:35,622 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:35,622 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:35,622 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:35,622 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:35,622 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,623 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:35,623 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:35,624 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,624 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,624 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,624 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:35,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:35,626 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:35,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@267a2e93[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:35,626 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,628 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,628 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:35,628 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:35,628 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:35,628 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,628 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:35,629 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,629 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:35,629 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,629 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:35,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:35,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:35,631 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:35,632 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:35,632 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:35,632 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:35,632 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:35,633 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,633 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,633 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,634 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,635 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,640 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,640 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,640 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,640 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:35,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:35,640 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,641 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,641 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,641 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:35,641 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:35,641 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,641 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,641 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:35,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54018 deadline: 1681890935738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,741 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:35,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:35,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:35,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890935739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:35,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:35,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:35,761 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:35,761 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:35,761 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:35,761 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:35,761 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,761 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,762 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,762 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,763 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,763 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:35,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:35,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:35,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:35,765 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:35,765 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:35,765 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:35,765 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:35,765 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:35,765 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:35,765 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:35,765 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,765 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:35,765 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:35,767 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,767 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,767 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,767 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:35,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@300f5ff5[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:35,768 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:35,769 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,770 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,770 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:35,770 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,770 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:35,770 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:35,770 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:35,772 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:35,772 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,772 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,772 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:35,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:35,775 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:35,775 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:35,775 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:35,775 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:35,775 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:35,777 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,777 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,777 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,778 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,787 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,790 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,790 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,790 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:35,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:35,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:35,790 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,790 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,790 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,790 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:35,790 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:35,790 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,790 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,790 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,790 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:35,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54044 deadline: 1681890935809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,813 DEBUG [Thread-863] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18211 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:35,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:35,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54054 deadline: 1681890935830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,831 DEBUG [Thread-859] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18228 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:35,890 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:35,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:35,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:35,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:35,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:35,894 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:35,894 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:35,894 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:35,894 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:35,894 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,894 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,896 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,896 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,896 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:35,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:35,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:35,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:35,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:35,897 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:35,897 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:35,897 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:35,897 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:35,897 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:35,897 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:35,898 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:35,898 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:35,898 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:35,898 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:35,898 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:35,898 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,898 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:35,898 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:35,900 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,900 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,901 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,901 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:35,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76ec26c9[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:35,902 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:35,902 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,903 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,903 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:35,904 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,904 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:35,905 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:35,905 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,905 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,905 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:35,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:35,908 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:35,908 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:35,908 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:35,908 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:35,908 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:35,909 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,909 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:35,910 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:35,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,910 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,911 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:35,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:35,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,914 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:35,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:35,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:35,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:35,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:35,915 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:35,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:35,915 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:35,915 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:35,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:35,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:35,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:35,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:35,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:35,920 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=374, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/6c9f04350c4c404da720d0b16cdaaa19 2023-04-19 07:54:35,931 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/bcc7897e738b4c2fba11bd5fccf9b615 2023-04-19 07:54:36,015 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:36,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:36,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:36,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:36,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:36,019 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:36,019 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:36,019 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:36,019 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:36,019 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,019 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,020 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,020 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:36,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:36,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:36,021 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,021 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:36,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:36,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:36,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:36,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:36,022 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:36,022 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:36,022 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:36,022 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:36,022 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:36,022 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:36,023 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:36,023 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:36,023 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:36,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:36,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:36,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:36,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:36,025 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:36,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:36,025 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:36,025 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,025 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:36,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:36,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:36,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:36,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62fc4fa1[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:36,027 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:36,027 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:36,029 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,029 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:36,031 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:36,031 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:36,031 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,031 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:36,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:36,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:36,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:36,033 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:36,033 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:36,033 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:36,033 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:36,033 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:36,035 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,035 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:36,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:36,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,035 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:36,036 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,037 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,041 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,041 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,041 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,041 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,041 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,041 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,041 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,041 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:36,041 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:36,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:36,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:36,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:36,042 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:36,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:36,042 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:36,042 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:36,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:36,142 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:36,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:36,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:36,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:36,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:36,145 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:36,145 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:36,145 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:36,145 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:36,146 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,146 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,148 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,148 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:36,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:36,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:36,149 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,149 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:36,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:36,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:36,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:36,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:36,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:36,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:36,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:36,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:36,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:36,150 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:36,150 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:36,150 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:36,151 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:36,151 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:36,151 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:36,151 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,151 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:36,151 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:36,152 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,152 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:36,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:36,153 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,153 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:36,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:36,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:36,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:36,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:36,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1585fbdb[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:36,154 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:36,154 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,156 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,156 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:36,156 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:36,156 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:36,156 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,156 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:36,157 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,157 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:36,157 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:36,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:36,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:36,157 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:36,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:36,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:36,161 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:36,161 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:36,161 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:36,161 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:36,161 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:36,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:36,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,164 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:36,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:36,164 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:36,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,165 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,167 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,167 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:36,167 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:36,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:36,168 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:36,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:36,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:36,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:36,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,168 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:36,168 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:36,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:36,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:36,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54018 deadline: 1681890936244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:36,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890936248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,268 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:36,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:36,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:36,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:36,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:36,272 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:36,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:36,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:36,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:36,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,273 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:36,273 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:36,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:36,274 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,274 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:36,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:36,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:36,275 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:36,275 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:36,275 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:36,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:36,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:36,276 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:36,276 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:36,276 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:36,276 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:36,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:36,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:36,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:36,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:36,277 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,277 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:36,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:36,278 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,278 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:36,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:36,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f3ffe69[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:36,279 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:36,279 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,281 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,281 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:36,281 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:36,281 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,281 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:36,281 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:36,283 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,283 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:36,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:36,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:36,283 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:36,283 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:36,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:36,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:36,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:36,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:36,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:36,285 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:36,285 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:36,285 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:36,285 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:36,285 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:36,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:36,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:36,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:36,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:36,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:36,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:36,288 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:36,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,289 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:36,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:36,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:36,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:36,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:36,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:36,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:36,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,291 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:36,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:36,292 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:36,292 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:36,352 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/483a0b13d1b34f3fa5c5513862579136 2023-04-19 07:54:36,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/6c9f04350c4c404da720d0b16cdaaa19 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/6c9f04350c4c404da720d0b16cdaaa19 2023-04-19 07:54:36,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/6c9f04350c4c404da720d0b16cdaaa19, entries=200, sequenceid=374, filesize=32.3 K 2023-04-19 07:54:36,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/bcc7897e738b4c2fba11bd5fccf9b615 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/bcc7897e738b4c2fba11bd5fccf9b615 2023-04-19 07:54:36,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/bcc7897e738b4c2fba11bd5fccf9b615, entries=150, sequenceid=374, filesize=11.9 K 2023-04-19 07:54:36,368 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/483a0b13d1b34f3fa5c5513862579136 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/483a0b13d1b34f3fa5c5513862579136 2023-04-19 07:54:36,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/483a0b13d1b34f3fa5c5513862579136, entries=150, sequenceid=374, filesize=11.9 K 2023-04-19 07:54:36,375 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 2117921701b6b45f9e01175de52f16a3 in 1281ms, sequenceid=374, compaction requested=true 2023-04-19 07:54:36,375 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:36,375 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:36,375 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 07:54:36,375 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-19 07:54:36,380 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-19 07:54:36,380 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-19 07:54:36,384 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 109800 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-19 07:54:36,384 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/C is initiating minor compaction (all files) 2023-04-19 07:54:36,384 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 241580 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-19 07:54:36,384 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/C in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:36,385 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/A is initiating minor compaction (all files) 2023-04-19 07:54:36,385 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/A in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:36,385 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/aaa9fe070ac543e38edf46e459ab950d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/75232f2a29ae423b83fdfcfcd3bec030, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/8664f4bcc2fc486ebe57d18b79cdd4bf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/10cb02bc1f7748a693fe505749420dc7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/5659a9d3d01e4851b009d56d3895e072, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/374ee43263ed419fa062d00053e9ce19, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/3093e9ab35864ac5b744829848028a62, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/87b67b8c7405443e923206b04cbc1f4f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/483a0b13d1b34f3fa5c5513862579136] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=107.2 K 2023-04-19 07:54:36,385 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/ebe0e8cda53f4904a9c436f8145f73b6, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/3ea3d93837894ff5a79bf264253e306b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/657e18982de34d869475d26e1a72d8d9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a4bf4faef7bb41c1bb81ea418c1529cc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/ee3c96969fcd4807bfee4b16c63e28a8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/c11ee6464acc442a8f28156bd3898c61, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/742a1f1e12eb4437831ff59350eec94e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/fca0d09aac894790bc6060fa20edad6c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/6c9f04350c4c404da720d0b16cdaaa19] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=235.9 K 2023-04-19 07:54:36,385 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting aaa9fe070ac543e38edf46e459ab950d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1681890865771 2023-04-19 07:54:36,386 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting ebe0e8cda53f4904a9c436f8145f73b6, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1681890865771 2023-04-19 07:54:36,386 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 75232f2a29ae423b83fdfcfcd3bec030, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681890865902 2023-04-19 07:54:36,386 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 3ea3d93837894ff5a79bf264253e306b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681890865902 2023-04-19 07:54:36,387 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 8664f4bcc2fc486ebe57d18b79cdd4bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1681890866539 2023-04-19 07:54:36,387 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 657e18982de34d869475d26e1a72d8d9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1681890866539 2023-04-19 07:54:36,387 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 10cb02bc1f7748a693fe505749420dc7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681890867176 2023-04-19 07:54:36,387 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting a4bf4faef7bb41c1bb81ea418c1529cc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681890867176 2023-04-19 07:54:36,388 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 5659a9d3d01e4851b009d56d3895e072, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681890868300 2023-04-19 07:54:36,388 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting ee3c96969fcd4807bfee4b16c63e28a8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681890868300 2023-04-19 07:54:36,388 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 374ee43263ed419fa062d00053e9ce19, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1681890869477 2023-04-19 07:54:36,389 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting c11ee6464acc442a8f28156bd3898c61, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1681890869477 2023-04-19 07:54:36,389 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 3093e9ab35864ac5b744829848028a62, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1681890870623 2023-04-19 07:54:36,389 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 742a1f1e12eb4437831ff59350eec94e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1681890870623 2023-04-19 07:54:36,389 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 87b67b8c7405443e923206b04cbc1f4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681890871766 2023-04-19 07:54:36,389 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting fca0d09aac894790bc6060fa20edad6c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681890871766 2023-04-19 07:54:36,390 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 483a0b13d1b34f3fa5c5513862579136, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681890873970 2023-04-19 07:54:36,390 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 6c9f04350c4c404da720d0b16cdaaa19, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681890873965 2023-04-19 07:54:36,392 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:36,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:36,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:36,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:36,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:36,397 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:36,397 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:36,397 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:36,397 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:36,397 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,397 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,400 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,400 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:36,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:36,400 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,400 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:36,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:36,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:36,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:36,402 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:36,402 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:36,402 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:36,402 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:36,402 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:36,404 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:36,404 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:36,404 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 07:54:36,404 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:36,404 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:36,405 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:36,405 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:36,405 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:36,405 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:36,443 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#C#compaction#172 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:36,459 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#A#compaction#171 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:36,478 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419a49e0f2c1e914198ace4024f4842efa7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419a49e0f2c1e914198ace4024f4842efa7 2023-04-19 07:54:36,499 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=390, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/f6dd838e98154c3c880fd3c45700f6ee 2023-04-19 07:54:36,507 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/14113424794d434f964f2fae17181ab7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/14113424794d434f964f2fae17181ab7 2023-04-19 07:54:36,514 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/A of 2117921701b6b45f9e01175de52f16a3 into 14113424794d434f964f2fae17181ab7(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:36,514 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:36,514 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/A, priority=7, startTime=1681890876375; duration=0sec 2023-04-19 07:54:36,514 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:36,514 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-19 07:54:36,517 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 109800 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-19 07:54:36,517 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/B is initiating minor compaction (all files) 2023-04-19 07:54:36,518 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/B in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:36,518 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/8b8b9bcd7f62406abf559efea7d15a79, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/448f25cee33a405ca9c8dea97557179e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/1586696512b74bcfa1b4294f2fb98339, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c62b7b8dd0d943bb986566917235bab1, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/b1e7f57489c34865ad4c32b543d4a5f9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c2be367a6fac41d09e134924e5221d46, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/235cac3a50304e6f855c069f1d123c97, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/6e977a4133a14e96b8ca32a6266895be, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/bcc7897e738b4c2fba11bd5fccf9b615] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=107.2 K 2023-04-19 07:54:36,518 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 8b8b9bcd7f62406abf559efea7d15a79, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1681890865771 2023-04-19 07:54:36,520 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 448f25cee33a405ca9c8dea97557179e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681890865902 2023-04-19 07:54:36,521 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 1586696512b74bcfa1b4294f2fb98339, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1681890866539 2023-04-19 07:54:36,521 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting c62b7b8dd0d943bb986566917235bab1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681890867176 2023-04-19 07:54:36,522 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting b1e7f57489c34865ad4c32b543d4a5f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681890868300 2023-04-19 07:54:36,522 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting c2be367a6fac41d09e134924e5221d46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1681890869477 2023-04-19 07:54:36,523 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 235cac3a50304e6f855c069f1d123c97, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1681890870623 2023-04-19 07:54:36,523 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 6e977a4133a14e96b8ca32a6266895be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681890871766 2023-04-19 07:54:36,524 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting bcc7897e738b4c2fba11bd5fccf9b615, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681890873970 2023-04-19 07:54:36,551 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#B#compaction#175 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:36,872 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/a6756cf2fc4c490ca397abe572f9aafb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/a6756cf2fc4c490ca397abe572f9aafb 2023-04-19 07:54:36,878 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/C of 2117921701b6b45f9e01175de52f16a3 into a6756cf2fc4c490ca397abe572f9aafb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:36,878 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:36,878 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/C, priority=7, startTime=1681890876375; duration=0sec 2023-04-19 07:54:36,878 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:36,943 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/19840ddbbe0645ad921620f1132d6031 2023-04-19 07:54:36,957 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/6cd7ea65b5144f7e8f61792049e45d2d 2023-04-19 07:54:36,962 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/f6dd838e98154c3c880fd3c45700f6ee as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/f6dd838e98154c3c880fd3c45700f6ee 2023-04-19 07:54:36,967 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/f6dd838e98154c3c880fd3c45700f6ee, entries=100, sequenceid=390, filesize=18.6 K 2023-04-19 07:54:36,969 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/19840ddbbe0645ad921620f1132d6031 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/19840ddbbe0645ad921620f1132d6031 2023-04-19 07:54:36,970 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/f34d4a09a2664fcf9657e12eeb301297 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/f34d4a09a2664fcf9657e12eeb301297 2023-04-19 07:54:36,974 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/19840ddbbe0645ad921620f1132d6031, entries=100, sequenceid=390, filesize=9.5 K 2023-04-19 07:54:36,975 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/6cd7ea65b5144f7e8f61792049e45d2d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/6cd7ea65b5144f7e8f61792049e45d2d 2023-04-19 07:54:36,977 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/B of 2117921701b6b45f9e01175de52f16a3 into f34d4a09a2664fcf9657e12eeb301297(size=12.6 K), total size for store is 22.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:36,977 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:36,977 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/B, priority=7, startTime=1681890876375; duration=0sec 2023-04-19 07:54:36,977 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:36,981 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/6cd7ea65b5144f7e8f61792049e45d2d, entries=100, sequenceid=390, filesize=9.5 K 2023-04-19 07:54:36,982 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 2117921701b6b45f9e01175de52f16a3 in 578ms, sequenceid=390, compaction requested=false 2023-04-19 07:54:36,982 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:36,982 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:36,982 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:36,982 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:36,982 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,983 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:36,983 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:36,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,984 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:36,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:36,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:36,985 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,985 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:36,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:36,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:36,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:36,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48724129[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:36,986 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:36,987 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,988 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:36,988 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:36,988 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:36,988 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,988 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:36,988 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:36,989 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,989 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:36,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:36,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:36,990 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:36,990 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:36,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:36,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:36,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:36,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:36,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:36,993 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:36,993 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:36,993 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:36,993 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:36,993 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:36,995 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:36,995 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,995 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:36,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:36,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:36,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:36,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:36,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:36,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,998 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:36,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:36,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:36,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,000 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,000 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,000 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:37,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:37,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:37,000 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,001 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:37,001 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:37,000 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:37,001 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:37,001 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:37,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:37,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:37,001 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,001 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,001 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:37,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:37,101 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:37,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:37,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:37,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:37,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:37,105 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:37,105 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:37,105 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:37,105 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:37,106 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,106 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,107 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,107 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:37,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:37,108 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,108 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:37,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:37,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:37,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:37,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:37,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:37,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:37,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:37,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:37,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:37,116 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:37,116 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:37,117 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:37,117 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:37,117 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:37,117 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:37,117 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,117 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:37,117 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:37,122 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:37,122 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,122 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:37,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:37,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:37,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48739ec1[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:37,125 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:37,125 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,126 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,126 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,126 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:37,126 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:37,126 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:37,126 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:37,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,128 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:37,128 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:37,128 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:37,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:37,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:37,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:37,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:37,131 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:37,131 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:37,131 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:37,131 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:37,131 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:37,132 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,132 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:37,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:37,132 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:37,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:37,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:37,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:37,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,134 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:37,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,136 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:37,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,139 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,139 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:37,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:37,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:37,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:37,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:37,139 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:37,139 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:37,139 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:37,139 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:37,139 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,139 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,139 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,139 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,139 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,139 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:37,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:37,239 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:37,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:37,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:37,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:37,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:37,243 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:37,243 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:37,243 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:37,243 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:37,244 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,244 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,245 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,245 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:37,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:37,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:37,246 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,246 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:37,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:37,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:37,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:37,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:37,247 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:37,247 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:37,247 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:37,256 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:37,256 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:37,256 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:37,256 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:37,256 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-19 07:54:37,257 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:37,257 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:37,257 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:37,257 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:37,257 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:37,257 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:37,293 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041950e5a177b5f14202ad886ec8f73ef62f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041950e5a177b5f14202ad886ec8f73ef62f 2023-04-19 07:54:37,306 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=398, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/8583f95e727940909336fb308a530c72 2023-04-19 07:54:37,322 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/07f0c43f69274fb8a577b3201f4c090c 2023-04-19 07:54:37,342 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/b88ba2e969ca46e9b3774cd80d3ce75a 2023-04-19 07:54:37,347 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/8583f95e727940909336fb308a530c72 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/8583f95e727940909336fb308a530c72 2023-04-19 07:54:37,353 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/8583f95e727940909336fb308a530c72, entries=50, sequenceid=398, filesize=11.7 K 2023-04-19 07:54:37,354 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/07f0c43f69274fb8a577b3201f4c090c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/07f0c43f69274fb8a577b3201f4c090c 2023-04-19 07:54:37,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:37,354 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:37,361 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/07f0c43f69274fb8a577b3201f4c090c, entries=50, sequenceid=398, filesize=7.1 K 2023-04-19 07:54:37,362 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/b88ba2e969ca46e9b3774cd80d3ce75a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b88ba2e969ca46e9b3774cd80d3ce75a 2023-04-19 07:54:37,367 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b88ba2e969ca46e9b3774cd80d3ce75a, entries=50, sequenceid=398, filesize=7.1 K 2023-04-19 07:54:37,367 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-19 07:54:37,368 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-19 07:54:37,369 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=67.09 KB/68700 for 2117921701b6b45f9e01175de52f16a3 in 113ms, sequenceid=398, compaction requested=true 2023-04-19 07:54:37,369 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:37,369 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:37,369 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:37,369 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:37,369 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,369 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:37,369 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:37,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:37,370 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-19 07:54:37,370 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:37,370 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:37,370 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:37,370 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:37,370 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:37,370 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:37,371 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,371 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:37,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:37,373 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,373 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:37,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:37,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:37,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7212ef37[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:37,374 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:37,374 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,376 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,376 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:37,376 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:37,376 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:37,379 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:37,379 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:37,379 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:37,379 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,380 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:37,380 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:37,380 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:37,380 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:37,380 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:37,380 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:37,380 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:37,382 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,382 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:37,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,382 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:37,383 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:37,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:37,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:37,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:37,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,384 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:37,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:37,388 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:37,388 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:37,388 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,388 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,388 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,388 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:37,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:37,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:37,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:37,388 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,388 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,388 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,388 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:37,388 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:37,388 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:37,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:37,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:37,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:37,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:37,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890937425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:37,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54018 deadline: 1681890937427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,489 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:37,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:37,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:37,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:37,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:37,493 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:37,493 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:37,493 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:37,494 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:37,494 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,494 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:37,495 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:37,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:37,496 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,496 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:37,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:37,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:37,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:37,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:37,497 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:37,497 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:37,497 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:37,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:37,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:37,498 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:37,498 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. started... 2023-04-19 07:54:37,498 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. as already flushing 2023-04-19 07:54:37,498 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:37,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:37,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:37,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:37,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:37,501 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,501 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:37,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:37,501 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,501 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:37,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:37,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:37,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@527e9baa[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:37,503 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:37,503 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,504 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,505 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:37,505 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:37,505 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:37,505 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,505 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:37,506 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:37,506 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,506 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:37,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:37,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:37,506 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:37,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:37,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:37,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:37,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:37,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:37,509 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:37,509 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:37,509 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:37,509 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:37,509 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:37,512 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,512 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:37,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:37,512 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:37,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:37,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:37,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,513 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:37,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:37,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,514 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:37,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:37,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,520 DEBUG [Thread-870] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x210d1008 to 127.0.0.1:58087 2023-04-19 07:54:37,520 DEBUG [Thread-870] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:37,520 DEBUG [Thread-874] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x508246cc to 127.0.0.1:58087 2023-04-19 07:54:37,520 DEBUG [Thread-874] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:37,521 DEBUG [Thread-876] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08795451 to 127.0.0.1:58087 2023-04-19 07:54:37,521 DEBUG [Thread-876] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:37,521 DEBUG [Thread-872] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00c2598b to 127.0.0.1:58087 2023-04-19 07:54:37,521 DEBUG [Thread-872] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:37,523 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,523 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:37,523 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:37,523 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:37,523 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:37,523 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:37,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:37,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:37,523 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:37,523 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,523 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:37,523 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:37,523 DEBUG [Thread-869] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:37,523 DEBUG [Thread-869] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:37,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:37,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:37,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:37,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890937530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:37,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54018 deadline: 1681890937534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,624 DEBUG [Thread-869] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:37,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:37,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:37,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890937732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:37,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54018 deadline: 1681890937735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:37,847 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304195dbc07057c504f55afa08c3a9f7b4a0d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195dbc07057c504f55afa08c3a9f7b4a0d 2023-04-19 07:54:38,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:38,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890938033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:38,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:38,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54018 deadline: 1681890938037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:38,253 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=413, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/a93c5ae8616448dca5a7655370d4ab3c 2023-04-19 07:54:38,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:38,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54058 deadline: 1681890938538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:38,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:38,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54018 deadline: 1681890938539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:38,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/84abe6004667497d8a8c3146cc22b87b 2023-04-19 07:54:39,079 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/b7de1022e8d4463ea468e09a736dfd49 2023-04-19 07:54:39,083 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/a93c5ae8616448dca5a7655370d4ab3c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a93c5ae8616448dca5a7655370d4ab3c 2023-04-19 07:54:39,087 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a93c5ae8616448dca5a7655370d4ab3c, entries=150, sequenceid=413, filesize=25.4 K 2023-04-19 07:54:39,088 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/84abe6004667497d8a8c3146cc22b87b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/84abe6004667497d8a8c3146cc22b87b 2023-04-19 07:54:39,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/84abe6004667497d8a8c3146cc22b87b, entries=150, sequenceid=413, filesize=11.9 K 2023-04-19 07:54:39,092 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/b7de1022e8d4463ea468e09a736dfd49 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b7de1022e8d4463ea468e09a736dfd49 2023-04-19 07:54:39,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b7de1022e8d4463ea468e09a736dfd49, entries=150, sequenceid=413, filesize=11.9 K 2023-04-19 07:54:39,097 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2117921701b6b45f9e01175de52f16a3 in 1727ms, sequenceid=413, compaction requested=true 2023-04-19 07:54:39,097 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:39,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:39,097 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:39,097 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:39,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:39,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:39,098 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 83728 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:39,099 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/A is initiating minor compaction (all files) 2023-04-19 07:54:39,099 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 42096 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:39,099 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/A in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:39,099 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/B is initiating minor compaction (all files) 2023-04-19 07:54:39,099 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/14113424794d434f964f2fae17181ab7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/f6dd838e98154c3c880fd3c45700f6ee, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/8583f95e727940909336fb308a530c72, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a93c5ae8616448dca5a7655370d4ab3c] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=81.8 K 2023-04-19 07:54:39,099 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/B in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:39,099 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/f34d4a09a2664fcf9657e12eeb301297, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/19840ddbbe0645ad921620f1132d6031, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/07f0c43f69274fb8a577b3201f4c090c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/84abe6004667497d8a8c3146cc22b87b] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=41.1 K 2023-04-19 07:54:39,099 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 14113424794d434f964f2fae17181ab7, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681890873970 2023-04-19 07:54:39,099 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting f34d4a09a2664fcf9657e12eeb301297, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681890873970 2023-04-19 07:54:39,099 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting f6dd838e98154c3c880fd3c45700f6ee, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681890875122 2023-04-19 07:54:39,100 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 19840ddbbe0645ad921620f1132d6031, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681890875122 2023-04-19 07:54:39,100 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 8583f95e727940909336fb308a530c72, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1681890877253 2023-04-19 07:54:39,100 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 07f0c43f69274fb8a577b3201f4c090c, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1681890877253 2023-04-19 07:54:39,100 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting a93c5ae8616448dca5a7655370d4ab3c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681890877362 2023-04-19 07:54:39,100 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 84abe6004667497d8a8c3146cc22b87b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681890877362 2023-04-19 07:54:39,108 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#B#compaction#184 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:39,112 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#A#compaction#183 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:39,123 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/ca2417e2c8b44484aa2914d45e077cee as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/ca2417e2c8b44484aa2914d45e077cee 2023-04-19 07:54:39,127 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/B of 2117921701b6b45f9e01175de52f16a3 into ca2417e2c8b44484aa2914d45e077cee(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:39,127 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:39,127 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/B, priority=12, startTime=1681890879097; duration=0sec 2023-04-19 07:54:39,128 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:39,128 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:39,129 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 42096 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:39,129 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 2117921701b6b45f9e01175de52f16a3/C is initiating minor compaction (all files) 2023-04-19 07:54:39,129 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2117921701b6b45f9e01175de52f16a3/C in TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:39,129 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/a6756cf2fc4c490ca397abe572f9aafb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/6cd7ea65b5144f7e8f61792049e45d2d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b88ba2e969ca46e9b3774cd80d3ce75a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b7de1022e8d4463ea468e09a736dfd49] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp, totalSize=41.1 K 2023-04-19 07:54:39,129 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting a6756cf2fc4c490ca397abe572f9aafb, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681890873970 2023-04-19 07:54:39,130 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 6cd7ea65b5144f7e8f61792049e45d2d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1681890875122 2023-04-19 07:54:39,130 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting b88ba2e969ca46e9b3774cd80d3ce75a, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1681890877253 2023-04-19 07:54:39,130 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting b7de1022e8d4463ea468e09a736dfd49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681890877362 2023-04-19 07:54:39,139 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 2117921701b6b45f9e01175de52f16a3#C#compaction#185 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:39,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:39,540 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-19 07:54:39,540 DEBUG [Thread-865] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24bd3872 to 127.0.0.1:58087 2023-04-19 07:54:39,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:39,541 DEBUG [Thread-865] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:39,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:39,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:39,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:39,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:39,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:39,544 DEBUG [Thread-861] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x55cbb115 to 127.0.0.1:58087 2023-04-19 07:54:39,544 DEBUG [Thread-861] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:39,549 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/cd3c0e0e6cf747109df3dcd24d697214 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/cd3c0e0e6cf747109df3dcd24d697214 2023-04-19 07:54:39,553 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/C of 2117921701b6b45f9e01175de52f16a3 into cd3c0e0e6cf747109df3dcd24d697214(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:39,554 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:39,554 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/C, priority=12, startTime=1681890879097; duration=0sec 2023-04-19 07:54:39,554 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:39,554 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304192f491012244943e49b223bb18e61a297 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192f491012244943e49b223bb18e61a297 2023-04-19 07:54:39,960 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=432, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/12e581fc03ee4b7f9735fa9a29c70eaa 2023-04-19 07:54:40,335 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/573319ff7119480f8859685c80d4cd0c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/573319ff7119480f8859685c80d4cd0c 2023-04-19 07:54:40,339 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2117921701b6b45f9e01175de52f16a3/A of 2117921701b6b45f9e01175de52f16a3 into 573319ff7119480f8859685c80d4cd0c(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-19 07:54:40,339 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:40,340 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3., storeName=2117921701b6b45f9e01175de52f16a3/A, priority=12, startTime=1681890879097; duration=1sec 2023-04-19 07:54:40,340 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:40,370 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/c7cf85bc61924f7498a88e60cc73c52a 2023-04-19 07:54:40,779 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=432 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/e969cec935a848c6b7c8b8e5efb7551a 2023-04-19 07:54:40,783 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/12e581fc03ee4b7f9735fa9a29c70eaa as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/12e581fc03ee4b7f9735fa9a29c70eaa 2023-04-19 07:54:40,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/12e581fc03ee4b7f9735fa9a29c70eaa, entries=150, sequenceid=432, filesize=25.4 K 2023-04-19 07:54:40,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/c7cf85bc61924f7498a88e60cc73c52a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c7cf85bc61924f7498a88e60cc73c52a 2023-04-19 07:54:40,792 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c7cf85bc61924f7498a88e60cc73c52a, entries=150, sequenceid=432, filesize=11.9 K 2023-04-19 07:54:40,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/e969cec935a848c6b7c8b8e5efb7551a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/e969cec935a848c6b7c8b8e5efb7551a 2023-04-19 07:54:40,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/e969cec935a848c6b7c8b8e5efb7551a, entries=150, sequenceid=432, filesize=11.9 K 2023-04-19 07:54:40,797 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=6.71 KB/6870 for 2117921701b6b45f9e01175de52f16a3 in 1257ms, sequenceid=432, compaction requested=false 2023-04-19 07:54:40,797 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:41,033 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-19 07:54:42,016 DEBUG [Thread-867] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7aba8e46 to 127.0.0.1:58087 2023-04-19 07:54:42,016 DEBUG [Thread-867] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:45,858 DEBUG [Thread-863] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a71edf0 to 127.0.0.1:58087 2023-04-19 07:54:45,859 DEBUG [Thread-863] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:45,863 DEBUG [Thread-859] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x399105f2 to 127.0.0.1:58087 2023-04-19 07:54:45,863 DEBUG [Thread-859] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 10 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 80 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 10 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 159 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(398): read 4875 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(398): read 4876 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(402): scanned 2051 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(403): verified 6153 rows 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(402): scanned 2057 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(403): verified 6171 rows 2023-04-19 07:54:45,864 INFO [Listener at localhost/42517] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-19 07:54:45,864 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a607873 to 127.0.0.1:58087 2023-04-19 07:54:45,864 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:54:45,866 INFO [Listener at localhost/42517] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-19 07:54:45,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-19 07:54:45,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:45,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-19 07:54:45,871 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890885871"}]},"ts":"1681890885871"} 2023-04-19 07:54:45,872 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-19 07:54:45,874 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-19 07:54:45,875 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2117921701b6b45f9e01175de52f16a3, UNASSIGN}] 2023-04-19 07:54:45,876 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2117921701b6b45f9e01175de52f16a3, UNASSIGN 2023-04-19 07:54:45,877 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=2117921701b6b45f9e01175de52f16a3, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:45,877 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890885877"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890885877"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890885877"}]},"ts":"1681890885877"} 2023-04-19 07:54:45,878 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:54:45,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-19 07:54:46,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:46,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2117921701b6b45f9e01175de52f16a3, disabling compactions & flushes 2023-04-19 07:54:46,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:46,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:46,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. after waiting 0 ms 2023-04-19 07:54:46,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:46,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2117921701b6b45f9e01175de52f16a3 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-19 07:54:46,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=A 2023-04-19 07:54:46,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:46,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=B 2023-04-19 07:54:46,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:46,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2117921701b6b45f9e01175de52f16a3, store=C 2023-04-19 07:54:46,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:46,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-19 07:54:46,444 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419cd9c0d40117d47abb21fa5373da707f7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419cd9c0d40117d47abb21fa5373da707f7 2023-04-19 07:54:46,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-19 07:54:46,849 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=441, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/89c2a11455c14e6893830aa2ee220ae8 2023-04-19 07:54:46,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-19 07:54:47,261 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/25518378bbe14d0cb28b934192f8cb37 2023-04-19 07:54:47,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/f67ab1f7543e47ad8946e6480c43a6bd 2023-04-19 07:54:47,674 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/A/89c2a11455c14e6893830aa2ee220ae8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/89c2a11455c14e6893830aa2ee220ae8 2023-04-19 07:54:47,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/89c2a11455c14e6893830aa2ee220ae8, entries=100, sequenceid=441, filesize=18.6 K 2023-04-19 07:54:47,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/B/25518378bbe14d0cb28b934192f8cb37 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/25518378bbe14d0cb28b934192f8cb37 2023-04-19 07:54:47,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/25518378bbe14d0cb28b934192f8cb37, entries=100, sequenceid=441, filesize=9.5 K 2023-04-19 07:54:47,683 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/.tmp/C/f67ab1f7543e47ad8946e6480c43a6bd as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/f67ab1f7543e47ad8946e6480c43a6bd 2023-04-19 07:54:47,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/f67ab1f7543e47ad8946e6480c43a6bd, entries=100, sequenceid=441, filesize=9.5 K 2023-04-19 07:54:47,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 2117921701b6b45f9e01175de52f16a3 in 1658ms, sequenceid=441, compaction requested=true 2023-04-19 07:54:47,706 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/b896c9e93e1d4b8e8704fe2dc76a142b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/41f330c6ecc54eafa3316f88fff75cf7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/69f35e41d5f0421ebc5e7c10d860b34a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/d8c331f744e547f9b5f6378b5a826512, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a36eace486cb44a8a90f6af9050f5b4d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/8b5c45880d9346ce9967079f46cf6537, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/af811d918a734f06876708092d57fccb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/5c77c9a0cb2e4081a4fdd2e7d4729551, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/6527752509744515a433a57f89a81c73, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a2db6155b7e94aec8c0a3e0a0ad4785a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/0f4fad54502b450c9c392a410039b502, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/ebe0e8cda53f4904a9c436f8145f73b6, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/3ea3d93837894ff5a79bf264253e306b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/657e18982de34d869475d26e1a72d8d9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a4bf4faef7bb41c1bb81ea418c1529cc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/ee3c96969fcd4807bfee4b16c63e28a8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/c11ee6464acc442a8f28156bd3898c61, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/742a1f1e12eb4437831ff59350eec94e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/fca0d09aac894790bc6060fa20edad6c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/6c9f04350c4c404da720d0b16cdaaa19, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/14113424794d434f964f2fae17181ab7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/f6dd838e98154c3c880fd3c45700f6ee, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/8583f95e727940909336fb308a530c72, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a93c5ae8616448dca5a7655370d4ab3c] to archive 2023-04-19 07:54:47,707 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:54:47,709 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/b896c9e93e1d4b8e8704fe2dc76a142b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/b896c9e93e1d4b8e8704fe2dc76a142b 2023-04-19 07:54:47,710 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/41f330c6ecc54eafa3316f88fff75cf7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/41f330c6ecc54eafa3316f88fff75cf7 2023-04-19 07:54:47,711 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/69f35e41d5f0421ebc5e7c10d860b34a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/69f35e41d5f0421ebc5e7c10d860b34a 2023-04-19 07:54:47,712 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/d8c331f744e547f9b5f6378b5a826512 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/d8c331f744e547f9b5f6378b5a826512 2023-04-19 07:54:47,713 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a36eace486cb44a8a90f6af9050f5b4d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a36eace486cb44a8a90f6af9050f5b4d 2023-04-19 07:54:47,714 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/8b5c45880d9346ce9967079f46cf6537 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/8b5c45880d9346ce9967079f46cf6537 2023-04-19 07:54:47,715 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/af811d918a734f06876708092d57fccb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/af811d918a734f06876708092d57fccb 2023-04-19 07:54:47,716 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/5c77c9a0cb2e4081a4fdd2e7d4729551 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/5c77c9a0cb2e4081a4fdd2e7d4729551 2023-04-19 07:54:47,717 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/6527752509744515a433a57f89a81c73 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/6527752509744515a433a57f89a81c73 2023-04-19 07:54:47,718 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a2db6155b7e94aec8c0a3e0a0ad4785a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a2db6155b7e94aec8c0a3e0a0ad4785a 2023-04-19 07:54:47,719 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/0f4fad54502b450c9c392a410039b502 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/0f4fad54502b450c9c392a410039b502 2023-04-19 07:54:47,720 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/ebe0e8cda53f4904a9c436f8145f73b6 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/ebe0e8cda53f4904a9c436f8145f73b6 2023-04-19 07:54:47,721 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/3ea3d93837894ff5a79bf264253e306b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/3ea3d93837894ff5a79bf264253e306b 2023-04-19 07:54:47,722 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/657e18982de34d869475d26e1a72d8d9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/657e18982de34d869475d26e1a72d8d9 2023-04-19 07:54:47,723 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a4bf4faef7bb41c1bb81ea418c1529cc to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a4bf4faef7bb41c1bb81ea418c1529cc 2023-04-19 07:54:47,724 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/ee3c96969fcd4807bfee4b16c63e28a8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/ee3c96969fcd4807bfee4b16c63e28a8 2023-04-19 07:54:47,725 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/c11ee6464acc442a8f28156bd3898c61 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/c11ee6464acc442a8f28156bd3898c61 2023-04-19 07:54:47,726 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/742a1f1e12eb4437831ff59350eec94e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/742a1f1e12eb4437831ff59350eec94e 2023-04-19 07:54:47,727 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/fca0d09aac894790bc6060fa20edad6c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/fca0d09aac894790bc6060fa20edad6c 2023-04-19 07:54:47,728 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/6c9f04350c4c404da720d0b16cdaaa19 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/6c9f04350c4c404da720d0b16cdaaa19 2023-04-19 07:54:47,729 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/14113424794d434f964f2fae17181ab7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/14113424794d434f964f2fae17181ab7 2023-04-19 07:54:47,730 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/f6dd838e98154c3c880fd3c45700f6ee to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/f6dd838e98154c3c880fd3c45700f6ee 2023-04-19 07:54:47,731 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/8583f95e727940909336fb308a530c72 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/8583f95e727940909336fb308a530c72 2023-04-19 07:54:47,732 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a93c5ae8616448dca5a7655370d4ab3c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/a93c5ae8616448dca5a7655370d4ab3c 2023-04-19 07:54:47,734 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fc3305f66c043af8312caf309122a2f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/dd1b9e6571fa4d58a0a8ea9bb87db77b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fc3e690b1b74c75831ecbbbf3827abd, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2c88b8d835f44af882a160638ab50d99, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fbba6039d094038b2104091e96e6aaf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/22fdc7d5aeed4b4d83fa41a4aba4d36b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/aa88b9a4ce9d4a0e8ca246b93b7a2108, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/f0f568b5a367498b88991e25c16cb58d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/769e3ea7607c482fabd8174336083cb3, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/eb33ebd2272747db8f8c95a362c4fc6b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/8b8b9bcd7f62406abf559efea7d15a79, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/b28f11b230d84ef5b60fa9f8e0d1349c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/448f25cee33a405ca9c8dea97557179e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/1586696512b74bcfa1b4294f2fb98339, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c62b7b8dd0d943bb986566917235bab1, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/b1e7f57489c34865ad4c32b543d4a5f9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c2be367a6fac41d09e134924e5221d46, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/235cac3a50304e6f855c069f1d123c97, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/6e977a4133a14e96b8ca32a6266895be, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/f34d4a09a2664fcf9657e12eeb301297, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/bcc7897e738b4c2fba11bd5fccf9b615, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/19840ddbbe0645ad921620f1132d6031, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/07f0c43f69274fb8a577b3201f4c090c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/84abe6004667497d8a8c3146cc22b87b] to archive 2023-04-19 07:54:47,735 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:54:47,736 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fc3305f66c043af8312caf309122a2f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fc3305f66c043af8312caf309122a2f 2023-04-19 07:54:47,737 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/dd1b9e6571fa4d58a0a8ea9bb87db77b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/dd1b9e6571fa4d58a0a8ea9bb87db77b 2023-04-19 07:54:47,739 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fc3e690b1b74c75831ecbbbf3827abd to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fc3e690b1b74c75831ecbbbf3827abd 2023-04-19 07:54:47,740 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2c88b8d835f44af882a160638ab50d99 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2c88b8d835f44af882a160638ab50d99 2023-04-19 07:54:47,741 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fbba6039d094038b2104091e96e6aaf to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/2fbba6039d094038b2104091e96e6aaf 2023-04-19 07:54:47,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/22fdc7d5aeed4b4d83fa41a4aba4d36b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/22fdc7d5aeed4b4d83fa41a4aba4d36b 2023-04-19 07:54:47,743 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/aa88b9a4ce9d4a0e8ca246b93b7a2108 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/aa88b9a4ce9d4a0e8ca246b93b7a2108 2023-04-19 07:54:47,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/f0f568b5a367498b88991e25c16cb58d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/f0f568b5a367498b88991e25c16cb58d 2023-04-19 07:54:47,745 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/769e3ea7607c482fabd8174336083cb3 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/769e3ea7607c482fabd8174336083cb3 2023-04-19 07:54:47,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/eb33ebd2272747db8f8c95a362c4fc6b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/eb33ebd2272747db8f8c95a362c4fc6b 2023-04-19 07:54:47,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/8b8b9bcd7f62406abf559efea7d15a79 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/8b8b9bcd7f62406abf559efea7d15a79 2023-04-19 07:54:47,748 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/b28f11b230d84ef5b60fa9f8e0d1349c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/b28f11b230d84ef5b60fa9f8e0d1349c 2023-04-19 07:54:47,749 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/448f25cee33a405ca9c8dea97557179e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/448f25cee33a405ca9c8dea97557179e 2023-04-19 07:54:47,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/1586696512b74bcfa1b4294f2fb98339 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/1586696512b74bcfa1b4294f2fb98339 2023-04-19 07:54:47,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c62b7b8dd0d943bb986566917235bab1 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c62b7b8dd0d943bb986566917235bab1 2023-04-19 07:54:47,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/b1e7f57489c34865ad4c32b543d4a5f9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/b1e7f57489c34865ad4c32b543d4a5f9 2023-04-19 07:54:47,754 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c2be367a6fac41d09e134924e5221d46 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c2be367a6fac41d09e134924e5221d46 2023-04-19 07:54:47,755 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/235cac3a50304e6f855c069f1d123c97 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/235cac3a50304e6f855c069f1d123c97 2023-04-19 07:54:47,756 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/6e977a4133a14e96b8ca32a6266895be to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/6e977a4133a14e96b8ca32a6266895be 2023-04-19 07:54:47,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/f34d4a09a2664fcf9657e12eeb301297 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/f34d4a09a2664fcf9657e12eeb301297 2023-04-19 07:54:47,758 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/bcc7897e738b4c2fba11bd5fccf9b615 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/bcc7897e738b4c2fba11bd5fccf9b615 2023-04-19 07:54:47,759 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/19840ddbbe0645ad921620f1132d6031 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/19840ddbbe0645ad921620f1132d6031 2023-04-19 07:54:47,760 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/07f0c43f69274fb8a577b3201f4c090c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/07f0c43f69274fb8a577b3201f4c090c 2023-04-19 07:54:47,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/84abe6004667497d8a8c3146cc22b87b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/84abe6004667497d8a8c3146cc22b87b 2023-04-19 07:54:47,764 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/7afd3c7c092e4135bc7e7e98aa804f45, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/3372db3442c740e38ef8c7af0ab12da2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/015d251b108d49d9983507072b0b3e0c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/de181201bf6e435689c0335a2290565b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/cea8ff20c43c4a919aa85167c6a01a35, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/ed3c4bddc09a40bda6b7cf57706f3410, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b5bc9937e978476eb0b7c9d32d2557cb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/e989072bca4c4282868456b81b544073, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/830cf20241e14bbd87de97101b38829b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/8e27891d701e4ed3b9507eb5ee8303fa, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/aaa9fe070ac543e38edf46e459ab950d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/22bc4040d80d4544b286524aff2d8253, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/75232f2a29ae423b83fdfcfcd3bec030, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/8664f4bcc2fc486ebe57d18b79cdd4bf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/10cb02bc1f7748a693fe505749420dc7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/5659a9d3d01e4851b009d56d3895e072, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/374ee43263ed419fa062d00053e9ce19, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/3093e9ab35864ac5b744829848028a62, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/87b67b8c7405443e923206b04cbc1f4f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/a6756cf2fc4c490ca397abe572f9aafb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/483a0b13d1b34f3fa5c5513862579136, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/6cd7ea65b5144f7e8f61792049e45d2d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b88ba2e969ca46e9b3774cd80d3ce75a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b7de1022e8d4463ea468e09a736dfd49] to archive 2023-04-19 07:54:47,765 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:54:47,767 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/7afd3c7c092e4135bc7e7e98aa804f45 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/7afd3c7c092e4135bc7e7e98aa804f45 2023-04-19 07:54:47,768 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/3372db3442c740e38ef8c7af0ab12da2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/3372db3442c740e38ef8c7af0ab12da2 2023-04-19 07:54:47,768 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/015d251b108d49d9983507072b0b3e0c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/015d251b108d49d9983507072b0b3e0c 2023-04-19 07:54:47,769 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/de181201bf6e435689c0335a2290565b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/de181201bf6e435689c0335a2290565b 2023-04-19 07:54:47,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/cea8ff20c43c4a919aa85167c6a01a35 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/cea8ff20c43c4a919aa85167c6a01a35 2023-04-19 07:54:47,771 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/ed3c4bddc09a40bda6b7cf57706f3410 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/ed3c4bddc09a40bda6b7cf57706f3410 2023-04-19 07:54:47,772 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b5bc9937e978476eb0b7c9d32d2557cb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b5bc9937e978476eb0b7c9d32d2557cb 2023-04-19 07:54:47,773 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/e989072bca4c4282868456b81b544073 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/e989072bca4c4282868456b81b544073 2023-04-19 07:54:47,774 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/830cf20241e14bbd87de97101b38829b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/830cf20241e14bbd87de97101b38829b 2023-04-19 07:54:47,775 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/8e27891d701e4ed3b9507eb5ee8303fa to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/8e27891d701e4ed3b9507eb5ee8303fa 2023-04-19 07:54:47,777 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/aaa9fe070ac543e38edf46e459ab950d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/aaa9fe070ac543e38edf46e459ab950d 2023-04-19 07:54:47,778 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/22bc4040d80d4544b286524aff2d8253 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/22bc4040d80d4544b286524aff2d8253 2023-04-19 07:54:47,779 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/75232f2a29ae423b83fdfcfcd3bec030 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/75232f2a29ae423b83fdfcfcd3bec030 2023-04-19 07:54:47,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/8664f4bcc2fc486ebe57d18b79cdd4bf to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/8664f4bcc2fc486ebe57d18b79cdd4bf 2023-04-19 07:54:47,781 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/10cb02bc1f7748a693fe505749420dc7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/10cb02bc1f7748a693fe505749420dc7 2023-04-19 07:54:47,782 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/5659a9d3d01e4851b009d56d3895e072 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/5659a9d3d01e4851b009d56d3895e072 2023-04-19 07:54:47,783 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/374ee43263ed419fa062d00053e9ce19 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/374ee43263ed419fa062d00053e9ce19 2023-04-19 07:54:47,784 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/3093e9ab35864ac5b744829848028a62 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/3093e9ab35864ac5b744829848028a62 2023-04-19 07:54:47,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/87b67b8c7405443e923206b04cbc1f4f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/87b67b8c7405443e923206b04cbc1f4f 2023-04-19 07:54:47,786 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/a6756cf2fc4c490ca397abe572f9aafb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/a6756cf2fc4c490ca397abe572f9aafb 2023-04-19 07:54:47,787 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/483a0b13d1b34f3fa5c5513862579136 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/483a0b13d1b34f3fa5c5513862579136 2023-04-19 07:54:47,788 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/6cd7ea65b5144f7e8f61792049e45d2d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/6cd7ea65b5144f7e8f61792049e45d2d 2023-04-19 07:54:47,788 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b88ba2e969ca46e9b3774cd80d3ce75a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b88ba2e969ca46e9b3774cd80d3ce75a 2023-04-19 07:54:47,790 DEBUG [StoreCloser-TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b7de1022e8d4463ea468e09a736dfd49 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/b7de1022e8d4463ea468e09a736dfd49 2023-04-19 07:54:47,794 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/recovered.edits/444.seqid, newMaxSeqId=444, maxSeqId=4 2023-04-19 07:54:47,795 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3. 2023-04-19 07:54:47,795 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2117921701b6b45f9e01175de52f16a3: 2023-04-19 07:54:47,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:47,797 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=2117921701b6b45f9e01175de52f16a3, regionState=CLOSED 2023-04-19 07:54:47,797 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890887797"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890887797"}]},"ts":"1681890887797"} 2023-04-19 07:54:47,799 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-19 07:54:47,799 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 2117921701b6b45f9e01175de52f16a3, server=jenkins-hbase4.apache.org,42359,1681890819191 in 1.9200 sec 2023-04-19 07:54:47,800 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-19 07:54:47,800 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2117921701b6b45f9e01175de52f16a3, UNASSIGN in 1.9240 sec 2023-04-19 07:54:47,801 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890887801"}]},"ts":"1681890887801"} 2023-04-19 07:54:47,802 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-19 07:54:47,804 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-19 07:54:47,805 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9380 sec 2023-04-19 07:54:47,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-19 07:54:47,975 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-19 07:54:47,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-19 07:54:47,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:47,978 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:47,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-19 07:54:47,978 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:47,983 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:47,985 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/recovered.edits] 2023-04-19 07:54:47,987 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/12e581fc03ee4b7f9735fa9a29c70eaa to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/12e581fc03ee4b7f9735fa9a29c70eaa 2023-04-19 07:54:47,988 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/573319ff7119480f8859685c80d4cd0c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/573319ff7119480f8859685c80d4cd0c 2023-04-19 07:54:47,989 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/89c2a11455c14e6893830aa2ee220ae8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/A/89c2a11455c14e6893830aa2ee220ae8 2023-04-19 07:54:47,992 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/25518378bbe14d0cb28b934192f8cb37 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/25518378bbe14d0cb28b934192f8cb37 2023-04-19 07:54:47,993 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c7cf85bc61924f7498a88e60cc73c52a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/c7cf85bc61924f7498a88e60cc73c52a 2023-04-19 07:54:47,994 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/ca2417e2c8b44484aa2914d45e077cee to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/B/ca2417e2c8b44484aa2914d45e077cee 2023-04-19 07:54:47,996 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/cd3c0e0e6cf747109df3dcd24d697214 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/cd3c0e0e6cf747109df3dcd24d697214 2023-04-19 07:54:47,997 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/e969cec935a848c6b7c8b8e5efb7551a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/e969cec935a848c6b7c8b8e5efb7551a 2023-04-19 07:54:47,999 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/f67ab1f7543e47ad8946e6480c43a6bd to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/C/f67ab1f7543e47ad8946e6480c43a6bd 2023-04-19 07:54:48,002 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/recovered.edits/444.seqid to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3/recovered.edits/444.seqid 2023-04-19 07:54:48,002 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/2117921701b6b45f9e01175de52f16a3 2023-04-19 07:54:48,003 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 07:54:48,003 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-19 07:54:48,004 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-19 07:54:48,008 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304190130c31e07f04009b1b8bc1b90f03e2b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304190130c31e07f04009b1b8bc1b90f03e2b 2023-04-19 07:54:48,010 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041904b269ba74284952af1c96db57084c03 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041904b269ba74284952af1c96db57084c03 2023-04-19 07:54:48,011 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191c880dd7a6164ccc868f39bb157ea734 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191c880dd7a6164ccc868f39bb157ea734 2023-04-19 07:54:48,013 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041922402447d06147aa82e0dee29347aa15 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041922402447d06147aa82e0dee29347aa15 2023-04-19 07:54:48,014 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419248cfddefbaa4c8a81f09f73b129e871 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419248cfddefbaa4c8a81f09f73b129e871 2023-04-19 07:54:48,015 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192671d145659542f4964b4b28d3e410f8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192671d145659542f4964b4b28d3e410f8 2023-04-19 07:54:48,016 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192f491012244943e49b223bb18e61a297 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192f491012244943e49b223bb18e61a297 2023-04-19 07:54:48,017 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194198a056d9bb41088c527bf0c86bf5d2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194198a056d9bb41088c527bf0c86bf5d2 2023-04-19 07:54:48,018 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041946f2cad00f8a4bc89a1979a9ed6a497b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041946f2cad00f8a4bc89a1979a9ed6a497b 2023-04-19 07:54:48,020 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194749f96db1e847c0964a02ceba342b4e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194749f96db1e847c0964a02ceba342b4e 2023-04-19 07:54:48,021 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419475775ce722744e4932d0b09915c381a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419475775ce722744e4932d0b09915c381a 2023-04-19 07:54:48,022 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194fda67768778496da08c95fbc1dd89bf to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194fda67768778496da08c95fbc1dd89bf 2023-04-19 07:54:48,023 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041950e5a177b5f14202ad886ec8f73ef62f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041950e5a177b5f14202ad886ec8f73ef62f 2023-04-19 07:54:48,024 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195dbc07057c504f55afa08c3a9f7b4a0d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195dbc07057c504f55afa08c3a9f7b4a0d 2023-04-19 07:54:48,025 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419614434a8fb6944658e413b29d9eb5199 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419614434a8fb6944658e413b29d9eb5199 2023-04-19 07:54:48,027 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197d53b0e81426495b92d39961241fc5c8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197d53b0e81426495b92d39961241fc5c8 2023-04-19 07:54:48,028 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419a49e0f2c1e914198ace4024f4842efa7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419a49e0f2c1e914198ace4024f4842efa7 2023-04-19 07:54:48,029 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b1996960feaa487f9ff1cedf33b50448 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b1996960feaa487f9ff1cedf33b50448 2023-04-19 07:54:48,030 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419cd9c0d40117d47abb21fa5373da707f7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419cd9c0d40117d47abb21fa5373da707f7 2023-04-19 07:54:48,031 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d1d0ffe5b2194564b573e2cf29ac02a3 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d1d0ffe5b2194564b573e2cf29ac02a3 2023-04-19 07:54:48,033 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d3022f449faa4f2eac22f813d70913de to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d3022f449faa4f2eac22f813d70913de 2023-04-19 07:54:48,034 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f84bd1c69a014a10a305c7c1c42faf5b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f84bd1c69a014a10a305c7c1c42faf5b 2023-04-19 07:54:48,034 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-19 07:54:48,037 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:48,038 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-19 07:54:48,040 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-19 07:54:48,041 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:48,041 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-19 07:54:48,041 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681890888041"}]},"ts":"9223372036854775807"} 2023-04-19 07:54:48,043 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-19 07:54:48,043 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2117921701b6b45f9e01175de52f16a3, NAME => 'TestAcidGuarantees,,1681890855377.2117921701b6b45f9e01175de52f16a3.', STARTKEY => '', ENDKEY => ''}] 2023-04-19 07:54:48,043 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-19 07:54:48,043 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681890888043"}]},"ts":"9223372036854775807"} 2023-04-19 07:54:48,044 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-19 07:54:48,046 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:48,047 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 71 msec 2023-04-19 07:54:48,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-19 07:54:48,079 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-19 07:54:48,091 INFO [Listener at localhost/42517] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 230), OpenFileDescriptor=535 (was 539), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=380 (was 424), ProcessCount=169 (was 169), AvailableMemoryMB=6258 (was 6407) 2023-04-19 07:54:48,105 INFO [Listener at localhost/42517] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=535, MaxFileDescriptor=60000, SystemLoadAverage=380, ProcessCount=169, AvailableMemoryMB=6257 2023-04-19 07:54:48,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 07:54:48,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 07:54:48,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-19 07:54:48,110 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-19 07:54:48,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-19 07:54:48,110 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-19 07:54:48,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-19 07:54:48,112 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:48,112 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c empty. 2023-04-19 07:54:48,113 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:48,113 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 07:54:48,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-19 07:54:48,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-19 07:54:48,522 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 07:54:48,523 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => cfaed3abed1fbb0d5a48467b790cb91c, NAME => 'TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp 2023-04-19 07:54:48,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-19 07:54:48,930 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:54:48,931 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing cfaed3abed1fbb0d5a48467b790cb91c, disabling compactions & flushes 2023-04-19 07:54:48,931 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:48,931 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:48,931 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. after waiting 0 ms 2023-04-19 07:54:48,931 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:48,931 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:48,931 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:48,934 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-19 07:54:48,935 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890888935"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890888935"}]},"ts":"1681890888935"} 2023-04-19 07:54:48,936 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-19 07:54:48,937 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-19 07:54:48,937 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890888937"}]},"ts":"1681890888937"} 2023-04-19 07:54:48,938 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-19 07:54:48,941 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cfaed3abed1fbb0d5a48467b790cb91c, ASSIGN}] 2023-04-19 07:54:48,943 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cfaed3abed1fbb0d5a48467b790cb91c, ASSIGN 2023-04-19 07:54:48,944 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=cfaed3abed1fbb0d5a48467b790cb91c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42359,1681890819191; forceNewPlan=false, retain=false 2023-04-19 07:54:49,095 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=cfaed3abed1fbb0d5a48467b790cb91c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:49,095 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890889095"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890889095"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890889095"}]},"ts":"1681890889095"} 2023-04-19 07:54:49,096 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:54:49,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-19 07:54:49,251 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:49,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cfaed3abed1fbb0d5a48467b790cb91c, NAME => 'TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.', STARTKEY => '', ENDKEY => ''} 2023-04-19 07:54:49,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:49,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:54:49,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:49,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:49,253 INFO [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:49,254 DEBUG [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A 2023-04-19 07:54:49,254 DEBUG [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A 2023-04-19 07:54:49,254 INFO [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:54:49,255 INFO [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cfaed3abed1fbb0d5a48467b790cb91c columnFamilyName A 2023-04-19 07:54:49,255 INFO [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] regionserver.HStore(310): Store=cfaed3abed1fbb0d5a48467b790cb91c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:54:49,255 INFO [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:49,256 DEBUG [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B 2023-04-19 07:54:49,256 DEBUG [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B 2023-04-19 07:54:49,256 INFO [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:54:49,257 INFO [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cfaed3abed1fbb0d5a48467b790cb91c columnFamilyName B 2023-04-19 07:54:49,257 INFO [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] regionserver.HStore(310): Store=cfaed3abed1fbb0d5a48467b790cb91c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:54:49,257 INFO [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:49,258 DEBUG [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C 2023-04-19 07:54:49,258 DEBUG [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C 2023-04-19 07:54:49,258 INFO [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:54:49,258 INFO [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cfaed3abed1fbb0d5a48467b790cb91c columnFamilyName C 2023-04-19 07:54:49,259 INFO [StoreOpener-cfaed3abed1fbb0d5a48467b790cb91c-1] regionserver.HStore(310): Store=cfaed3abed1fbb0d5a48467b790cb91c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:54:49,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:49,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:49,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:49,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 07:54:49,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:49,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 07:54:49,266 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened cfaed3abed1fbb0d5a48467b790cb91c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65649446, jitterRate=-0.02174702286720276}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 07:54:49,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:49,267 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., pid=30, masterSystemTime=1681890889248 2023-04-19 07:54:49,268 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:49,268 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:49,269 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=cfaed3abed1fbb0d5a48467b790cb91c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:49,269 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890889269"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681890889269"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681890889269"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681890889269"}]},"ts":"1681890889269"} 2023-04-19 07:54:49,271 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-19 07:54:49,271 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 in 174 msec 2023-04-19 07:54:49,272 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-19 07:54:49,273 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=cfaed3abed1fbb0d5a48467b790cb91c, ASSIGN in 330 msec 2023-04-19 07:54:49,273 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-19 07:54:49,273 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890889273"}]},"ts":"1681890889273"} 2023-04-19 07:54:49,274 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-19 07:54:49,277 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-19 07:54:49,278 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1690 sec 2023-04-19 07:54:50,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-19 07:54:50,215 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-19 07:54:50,216 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x458eac52 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:50,221 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7233cccd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:50,222 DEBUG [Listener at localhost/42517] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:50,223 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47200, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:50,224 DEBUG [Listener at localhost/42517] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-19 07:54:50,226 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37202, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-19 07:54:50,227 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x22b4eae5 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:50,231 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12e82edd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:50,231 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x262f4570 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:50,234 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23c97092, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:50,235 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x00ca898c to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:50,239 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c124820, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:50,240 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x6585a70a to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:50,243 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3157d4df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:50,243 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x7480f9da to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:50,246 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cf5bdb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:50,246 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x6919b2a0 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:50,250 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7810a231, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:50,250 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x2ce5573e to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:50,255 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8cd7cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:50,255 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x087b3095 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:50,258 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ea1ca03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:50,258 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x12115052 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:50,262 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@433f251d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:50,262 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x7e65578e to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:54:50,265 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66ee0bd0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:54:50,269 DEBUG [hconnection-0x1ccd5db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:50,270 DEBUG [hconnection-0x29fdf911-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:50,270 DEBUG [hconnection-0x5b6c3c6a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:50,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:50,271 DEBUG [hconnection-0x6f384ad3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:50,271 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47220, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:50,271 DEBUG [hconnection-0x75b83be6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:50,271 DEBUG [hconnection-0x39d15dcb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:50,272 DEBUG [hconnection-0x590a4dd2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:50,273 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47256, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:50,277 DEBUG [hconnection-0x65f5c321-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:50,277 DEBUG [hconnection-0x51ea305e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:50,278 DEBUG [hconnection-0x4d71a281-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:54:50,278 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47280, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:50,286 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47214, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:50,286 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47244, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:50,286 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47274, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:50,286 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47202, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:50,287 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47236, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:50,287 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47268, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:50,288 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47286, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:54:50,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:50,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:50,290 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:50,290 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:50,290 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:50,290 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:50,290 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,290 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:50,293 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,293 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:50,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:50,296 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,296 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:50,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:50,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:50,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:50,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:50,298 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:50,298 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:50,299 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-19 07:54:50,300 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:54:50,300 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:50,300 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:54:50,300 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:50,300 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:54:50,300 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:50,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:50,303 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:50,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890890311, max timestamp=1681890890319Num uniques 859; 2023-04-19 07:54:50,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890890311, max timestamp=1681890890319Num uniques 859; 2023-04-19 07:54:50,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:54:50,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:54:50,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890890311, max timestamp=1681890890319Num uniques 859; 2023-04-19 07:54:50,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:50,327 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/990964db056a47ebbfdf93750f2bda4d 2023-04-19 07:54:50,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890950336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47286 deadline: 1681890950337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890890331, max timestamp=1681890890339Num uniques 1068; 2023-04-19 07:54:50,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890890331, max timestamp=1681890890339Num uniques 1068; 2023-04-19 07:54:50,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-19 07:54:50,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-19 07:54:50,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890890331, max timestamp=1681890890339Num uniques 1018; 2023-04-19 07:54:50,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 07:54:50,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47236 deadline: 1681890950368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890890362, max timestamp=1681890890371Num uniques 1127; 2023-04-19 07:54:50,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890890362, max timestamp=1681890890371Num uniques 1127; 2023-04-19 07:54:50,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-19 07:54:50,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-19 07:54:50,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890890362, max timestamp=1681890890371Num uniques 1127; 2023-04-19 07:54:50,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-19 07:54:50,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47274 deadline: 1681890950403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681890890397, max timestamp=1681890890404Num uniques 1086; 2023-04-19 07:54:50,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681890890397, max timestamp=1681890890404Num uniques 1086; 2023-04-19 07:54:50,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-19 07:54:50,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-19 07:54:50,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681890890397, max timestamp=1681890890404Num uniques 1086; 2023-04-19 07:54:50,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-19 07:54:50,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681890890442, max timestamp=1681890890447Num uniques 1045; 2023-04-19 07:54:50,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681890890442, max timestamp=1681890890447Num uniques 1045; 2023-04-19 07:54:50,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681890890442, max timestamp=1681890890447Num uniques 1045; 2023-04-19 07:54:50,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-19 07:54:50,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-19 07:54:50,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-19 07:54:50,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890950472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681890890466, max timestamp=1681890890472Num uniques 1104; 2023-04-19 07:54:50,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681890890466, max timestamp=1681890890472Num uniques 1104; 2023-04-19 07:54:50,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47286 deadline: 1681890950473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-19 07:54:50,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681890890466, max timestamp=1681890890472Num uniques 1104; 2023-04-19 07:54:50,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-19 07:54:50,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-19 07:54:50,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7455, totalHeapSize=17192, min timestamp=1681890890527, max timestamp=1681890890534Num uniques 1063; 2023-04-19 07:54:50,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48675, new segment data size=7455, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-19 07:54:50,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681890890527, max timestamp=1681890890534Num uniques 1063; 2023-04-19 07:54:50,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681890890527, max timestamp=1681890890534Num uniques 1013; 2023-04-19 07:54:50,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-19 07:54:50,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-19 07:54:50,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681890890549, max timestamp=1681890890552Num uniques 1022; 2023-04-19 07:54:50,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681890890549, max timestamp=1681890890552Num uniques 1022; 2023-04-19 07:54:50,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-19 07:54:50,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-19 07:54:50,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681890890549, max timestamp=1681890890552Num uniques 1022; 2023-04-19 07:54:50,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-19 07:54:50,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47274 deadline: 1681890950569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681890890563, max timestamp=1681890890570Num uniques 1081; 2023-04-19 07:54:50,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681890890563, max timestamp=1681890890570Num uniques 1081; 2023-04-19 07:54:50,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681890890563, max timestamp=1681890890570Num uniques 1081; 2023-04-19 07:54:50,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-19 07:54:50,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-19 07:54:50,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-19 07:54:50,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1681890890580, max timestamp=1681890890613Num uniques 1090; 2023-04-19 07:54:50,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=15570, suffix heap size=141328,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=340 2023-04-19 07:54:50,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1681890890580, max timestamp=1681890890613Num uniques 1090; 2023-04-19 07:54:50,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=15570, suffix heap size=141328,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=340 2023-04-19 07:54:50,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1681890890580, max timestamp=1681890890613Num uniques 1090; 2023-04-19 07:54:50,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=15570, suffix heap size=141328,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=340 2023-04-19 07:54:50,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890950627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47236 deadline: 1681890950627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47286 deadline: 1681890950629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681890890624, max timestamp=1681890890629Num uniques 1199; 2023-04-19 07:54:50,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=9114, suffix heap size=152664,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=199 2023-04-19 07:54:50,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681890890624, max timestamp=1681890890629Num uniques 1199; 2023-04-19 07:54:50,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681890890624, max timestamp=1681890890629Num uniques 1199; 2023-04-19 07:54:50,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=9114, suffix heap size=152664,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=199 2023-04-19 07:54:50,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=9114, suffix heap size=152664,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=199 2023-04-19 07:54:50,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681890890664, max timestamp=1681890890675Num uniques 1058; 2023-04-19 07:54:50,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681890890664, max timestamp=1681890890675Num uniques 1058; 2023-04-19 07:54:50,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-19 07:54:50,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681890890664, max timestamp=1681890890675Num uniques 1058; 2023-04-19 07:54:50,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-19 07:54:50,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-19 07:54:50,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681890890690, max timestamp=1681890890696Num uniques 1067; 2023-04-19 07:54:50,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-19 07:54:50,699 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681890890690, max timestamp=1681890890696Num uniques 1067; 2023-04-19 07:54:50,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681890890690, max timestamp=1681890890696Num uniques 1067; 2023-04-19 07:54:50,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-19 07:54:50,700 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-19 07:54:50,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47244 deadline: 1681890950722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681890890718, max timestamp=1681890890723Num uniques 1076; 2023-04-19 07:54:50,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-19 07:54:50,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47274 deadline: 1681890950724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681890890718, max timestamp=1681890890723Num uniques 1076; 2023-04-19 07:54:50,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-19 07:54:50,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681890890718, max timestamp=1681890890723Num uniques 1076; 2023-04-19 07:54:50,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-19 07:54:50,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890950747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681890890742, max timestamp=1681890890748Num uniques 1035; 2023-04-19 07:54:50,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-19 07:54:50,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681890890745, max timestamp=1681890890748Num uniques 1035; 2023-04-19 07:54:50,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681890890745, max timestamp=1681890890748Num uniques 1035; 2023-04-19 07:54:50,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-19 07:54:50,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-19 07:54:50,763 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/7128a63e249b4b4192e8b679e4c9a146 2023-04-19 07:54:50,774 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/51bf9d43fbe84353b20a06297fd7105d 2023-04-19 07:54:50,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681890890768, max timestamp=1681890890775Num uniques 1094; 2023-04-19 07:54:50,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-19 07:54:50,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681890890768, max timestamp=1681890890775Num uniques 1044; 2023-04-19 07:54:50,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-19 07:54:50,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681890890768, max timestamp=1681890890775Num uniques 1044; 2023-04-19 07:54:50,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=6870, suffix heap size=136544,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=150 2023-04-19 07:54:50,780 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/990964db056a47ebbfdf93750f2bda4d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/990964db056a47ebbfdf93750f2bda4d 2023-04-19 07:54:50,786 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/990964db056a47ebbfdf93750f2bda4d, entries=100, sequenceid=9, filesize=9.3 K 2023-04-19 07:54:50,787 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/7128a63e249b4b4192e8b679e4c9a146 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7128a63e249b4b4192e8b679e4c9a146 2023-04-19 07:54:50,792 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7128a63e249b4b4192e8b679e4c9a146, entries=100, sequenceid=9, filesize=9.3 K 2023-04-19 07:54:50,793 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/51bf9d43fbe84353b20a06297fd7105d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/51bf9d43fbe84353b20a06297fd7105d 2023-04-19 07:54:50,796 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/51bf9d43fbe84353b20a06297fd7105d, entries=100, sequenceid=9, filesize=9.3 K 2023-04-19 07:54:50,797 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=103.14 KB/105616 for cfaed3abed1fbb0d5a48467b790cb91c in 498ms, sequenceid=9, compaction requested=false 2023-04-19 07:54:50,798 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:50,798 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:50,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:50,800 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,800 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:50,801 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,801 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:50,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:50,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:50,801 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=109.85 KB heapSize=307.70 KB 2023-04-19 07:54:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:54:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:50,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:50,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:54:50,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:50,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:54:50,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:50,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@85ff3c5[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:50,802 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:50,802 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:50,804 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,804 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:50,804 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:50,804 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:50,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:50,806 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:50,806 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:50,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:50,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:50,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:50,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:50,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:50,809 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:50,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:50,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:50,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:50,809 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:50,812 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,812 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:50,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,812 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:50,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:50,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:50,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,814 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:50,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:50,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:50,818 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:50,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:50,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:50,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:50,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:50,818 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:50,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:50,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:50,818 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:50,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:50,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47236 deadline: 1681890950821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47286 deadline: 1681890950822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47244 deadline: 1681890950826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47274 deadline: 1681890950826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890950852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,918 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:50,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:50,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:50,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:50,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:50,922 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:50,922 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:50,922 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:50,922 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:50,923 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,923 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:50,924 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:50,924 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:50,925 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,925 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:50,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:50,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:50,926 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:50,926 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:50,926 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:50,926 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:50,926 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:50,926 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:50,927 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:50,927 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:50,927 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:50,927 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,927 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:50,927 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:50,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,928 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:50,929 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,929 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:50,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47286 deadline: 1681890950928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:50,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:50,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47236 deadline: 1681890950928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:50,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:50,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0ef199[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:50,930 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:50,931 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,932 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,932 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:50,933 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,933 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:50,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:50,934 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:50,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:50,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:50,937 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:50,937 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:50,937 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:50,937 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:50,937 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:50,939 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,939 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:50,939 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:50,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:50,940 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,944 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:50,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,946 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:50,946 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:50,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:50,946 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,947 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:50,947 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:50,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:50,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:50,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:50,947 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:50,947 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:50,947 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:50,947 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:50,947 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,947 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:50,947 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:50,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:51,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47244 deadline: 1681890951027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47274 deadline: 1681890951027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,047 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:51,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:51,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:51,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:51,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:51,051 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:51,051 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:51,051 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:51,051 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:51,051 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,051 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,054 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:51,054 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:51,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:51,055 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,055 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:51,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:51,056 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:51,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:51,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:51,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890951056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,057 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:51,057 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:51,057 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:51,057 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:51,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:51,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:51,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:51,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:51,061 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,061 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,062 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:51,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3980c678[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:51,063 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:51,063 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,064 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:51,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:51,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:51,065 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,065 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:51,067 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:51,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:51,070 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:51,070 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:51,070 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:51,070 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:51,070 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:51,071 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,071 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:51,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,071 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,072 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,073 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,075 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,075 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,075 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,075 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,075 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,075 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,075 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,075 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:51,075 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:51,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:51,076 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:51,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:51,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:51,076 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:51,076 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:51,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:51,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:51,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47286 deadline: 1681890951131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47236 deadline: 1681890951132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,176 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:51,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:51,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:51,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:51,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:51,180 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:51,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:51,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:51,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:51,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,182 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,182 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:51,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:51,182 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,182 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:51,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:51,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:51,183 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:51,183 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:51,183 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:51,183 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:51,183 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:51,186 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,186 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,186 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,186 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:51,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f05b929[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:51,187 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:51,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,189 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,189 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:51,189 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:51,189 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:51,189 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,189 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:51,191 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,191 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:51,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:51,193 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:51,193 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:51,193 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:51,193 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:51,193 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:51,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,196 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:51,196 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,197 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,201 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,201 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,201 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:51,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:51,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:51,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:51,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:51,201 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:51,201 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:51,201 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,201 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:51,202 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,202 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:51,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:51,202 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:51,202 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,202 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/6b8487b5f6614f8fb38de11a9d6891a9 2023-04-19 07:54:51,302 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:51,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:51,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:51,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:51,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:51,306 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:51,306 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:51,306 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:51,306 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:51,306 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,306 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,308 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:51,308 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:51,308 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,308 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:51,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:51,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:51,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:51,310 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:51,310 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:51,310 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:51,310 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:51,312 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,312 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68b3b0d7[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:51,314 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:51,314 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,315 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,315 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:51,315 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:51,315 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:51,315 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,315 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:51,317 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,317 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:51,317 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,317 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:51,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:51,320 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:51,320 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:51,320 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:51,320 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:51,320 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:51,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:51,322 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,323 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,325 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:51,325 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:51,325 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,325 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,325 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:51,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:51,325 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:51,325 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,326 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,326 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:51,326 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:51,325 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:51,326 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:51,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:51,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47244 deadline: 1681890951331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47274 deadline: 1681890951333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890951359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,426 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:51,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:51,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:51,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:51,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:51,430 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:51,430 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:51,430 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:51,430 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:51,430 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,430 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,432 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:51,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,433 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:51,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:51,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47286 deadline: 1681890951433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47236 deadline: 1681890951435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:51,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:51,441 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:51,441 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:51,441 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:51,441 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:51,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:51,444 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,444 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,444 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,444 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:51,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@431a1661[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:51,448 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:51,449 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,452 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,452 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:51,452 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,453 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:51,453 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:51,453 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:51,454 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,454 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:51,455 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:51,455 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:51,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:51,457 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:51,457 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:51,457 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:51,458 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:51,458 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:51,460 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,460 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,460 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,462 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,463 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,466 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:51,466 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:51,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:51,466 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:51,466 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:51,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:51,466 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:51,466 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,467 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,467 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,467 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:51,467 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:51,467 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,467 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:51,567 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:51,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:51,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:51,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:51,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:51,570 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:51,571 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:51,571 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:51,571 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:51,571 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,571 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,573 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:51,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:51,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:51,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:51,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:51,575 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:51,575 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:51,575 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:51,575 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:51,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:51,577 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,577 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,577 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,577 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:51,579 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:51,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c16bb7c[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:51,579 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,581 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,581 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:51,581 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:51,582 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:51,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:51,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:51,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:51,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:51,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:51,586 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:51,586 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:51,586 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:51,586 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:51,586 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:51,588 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,588 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:51,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,588 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:51,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:51,588 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,590 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:51,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,592 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:51,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:51,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:51,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:51,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:51,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,592 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:51,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:51,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:51,592 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:51,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:51,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:51,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:51,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/66841a259cd842db8ef92f44cf587df7 2023-04-19 07:54:51,644 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=36.05 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/dff4b51069a4486ab11f8d3aa6cd9411 2023-04-19 07:54:51,648 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/6b8487b5f6614f8fb38de11a9d6891a9 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/6b8487b5f6614f8fb38de11a9d6891a9 2023-04-19 07:54:51,652 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/6b8487b5f6614f8fb38de11a9d6891a9, entries=150, sequenceid=300, filesize=11.9 K 2023-04-19 07:54:51,653 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/66841a259cd842db8ef92f44cf587df7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/66841a259cd842db8ef92f44cf587df7 2023-04-19 07:54:51,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/66841a259cd842db8ef92f44cf587df7, entries=150, sequenceid=300, filesize=11.9 K 2023-04-19 07:54:51,660 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/dff4b51069a4486ab11f8d3aa6cd9411 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/dff4b51069a4486ab11f8d3aa6cd9411 2023-04-19 07:54:51,682 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/dff4b51069a4486ab11f8d3aa6cd9411, entries=150, sequenceid=300, filesize=11.9 K 2023-04-19 07:54:51,683 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~116.56 KB/119356, heapSize ~327.58 KB/335440, currentSize=67.09 KB/68700 for cfaed3abed1fbb0d5a48467b790cb91c in 882ms, sequenceid=300, compaction requested=false 2023-04-19 07:54:51,683 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:51,692 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:51,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:51,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:51,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:51,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:51,700 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:51,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:51,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:51,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:51,701 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,701 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:51,703 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,704 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,704 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:51,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:51,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:51,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:51,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:51,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:51,705 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:51,705 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:51,705 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:51,712 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:51,712 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:51,712 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:51,712 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:51,712 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-19 07:54:51,712 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:54:51,712 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:51,712 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:54:51,713 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:51,713 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:54:51,713 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:51,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:51,837 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:51,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47244 deadline: 1681890951860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47274 deadline: 1681890951861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890951864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47286 deadline: 1681890951939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47236 deadline: 1681890951942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47244 deadline: 1681890951964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:51,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:51,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47274 deadline: 1681890951964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:52,132 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/1d5ad41481224c9a84c2b0e91b0058bf 2023-04-19 07:54:52,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:52,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:52,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47274 deadline: 1681890952167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:52,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890952168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:52,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:52,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890952471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:52,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:52,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47274 deadline: 1681890952471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:52,555 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/9edbb119ada84340ba2e4f7824b91381 2023-04-19 07:54:52,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:52,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890952866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:52,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:52,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47286 deadline: 1681890952948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:52,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:52,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47236 deadline: 1681890952953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:52,973 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/cf79064f7b364df8838a7c129ff79e96 2023-04-19 07:54:52,978 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/1d5ad41481224c9a84c2b0e91b0058bf as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/1d5ad41481224c9a84c2b0e91b0058bf 2023-04-19 07:54:52,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:52,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890952978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:52,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:52,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47274 deadline: 1681890952978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:52,983 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/1d5ad41481224c9a84c2b0e91b0058bf, entries=150, sequenceid=313, filesize=11.9 K 2023-04-19 07:54:52,984 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/9edbb119ada84340ba2e4f7824b91381 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/9edbb119ada84340ba2e4f7824b91381 2023-04-19 07:54:52,990 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/9edbb119ada84340ba2e4f7824b91381, entries=150, sequenceid=313, filesize=11.9 K 2023-04-19 07:54:52,991 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/cf79064f7b364df8838a7c129ff79e96 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cf79064f7b364df8838a7c129ff79e96 2023-04-19 07:54:52,995 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cf79064f7b364df8838a7c129ff79e96, entries=150, sequenceid=313, filesize=11.9 K 2023-04-19 07:54:52,995 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for cfaed3abed1fbb0d5a48467b790cb91c in 1283ms, sequenceid=313, compaction requested=true 2023-04-19 07:54:52,996 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:52,996 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:52,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:52,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:52,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:52,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:52,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:52,997 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:52,997 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:52,998 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:52,998 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:52,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:52,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e688926[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:52,999 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:52,999 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:53,000 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:53,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:53,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:53,000 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:53,000 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:53,000 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:53,000 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,000 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:53,001 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:53,002 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:53,001 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,002 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:53,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:53,004 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:53,004 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:53,004 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:53,004 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:53,004 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:53,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:53,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:53,007 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:53,008 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,010 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,011 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:53,011 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:53,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:53,011 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:53,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:53,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:53,011 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:53,011 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:53,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:53,011 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:53,011 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:53,011 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:53,011 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:53,011 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:53,011 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,111 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:53,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:53,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:53,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:53,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:53,115 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:53,115 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:53,115 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:53,115 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:53,115 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,115 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:53,116 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:53,116 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:53,117 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,117 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:53,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:53,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:53,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:53,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:53,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:53,119 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:53,119 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:53,119 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:53,119 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:53,119 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-19 07:54:53,119 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:54:53,119 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:53,119 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:54:53,119 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:53,119 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:54:53,119 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:53,332 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-19 07:54:53,527 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/2976b925835d494dacfbd460bfa387f4 2023-04-19 07:54:53,939 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/f20233317cda4a4c880978d72831f676 2023-04-19 07:54:53,950 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/c42cac8819944f3b925e047944c9fc34 2023-04-19 07:54:53,956 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/2976b925835d494dacfbd460bfa387f4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/2976b925835d494dacfbd460bfa387f4 2023-04-19 07:54:53,960 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/2976b925835d494dacfbd460bfa387f4, entries=150, sequenceid=332, filesize=11.9 K 2023-04-19 07:54:53,960 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/f20233317cda4a4c880978d72831f676 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/f20233317cda4a4c880978d72831f676 2023-04-19 07:54:53,964 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/f20233317cda4a4c880978d72831f676, entries=150, sequenceid=332, filesize=11.9 K 2023-04-19 07:54:53,965 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/c42cac8819944f3b925e047944c9fc34 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/c42cac8819944f3b925e047944c9fc34 2023-04-19 07:54:53,968 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/c42cac8819944f3b925e047944c9fc34, entries=150, sequenceid=332, filesize=11.9 K 2023-04-19 07:54:53,969 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for cfaed3abed1fbb0d5a48467b790cb91c in 850ms, sequenceid=332, compaction requested=true 2023-04-19 07:54:53,969 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:53,969 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:53,969 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:53,969 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:53,969 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,972 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:53,972 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:53,974 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:53,974 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:53,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:53,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b28ad34[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:53,976 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:53,976 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:53,978 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:53,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:53,979 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:53,979 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,979 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:53,983 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:53,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:53,985 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:53,985 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:53,985 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:53,985 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:53,985 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:53,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:53,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,987 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:53,988 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:53,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,989 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,994 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:53,994 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:53,994 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:53,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:53,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:53,994 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:53,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:53,994 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:53,994 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:53,994 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:53,994 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:53,994 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:53,994 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:53,994 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:53,994 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:53,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:53,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:53,998 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:54:53,998 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:54:53,999 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:53,999 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:54:53,999 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:53,999 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:54:53,999 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:54,016 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/12f00c3c56c94c1e984cb229f8e34f6e 2023-04-19 07:54:54,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890894020, max timestamp=1681890894039Num uniques 859; 2023-04-19 07:54:54,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:54:54,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681890894020, max timestamp=1681890894039Num uniques 859; 2023-04-19 07:54:54,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:54:54,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681890894020, max timestamp=1681890894039Num uniques 859; 2023-04-19 07:54:54,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:54:54,047 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/7b88c50217604ea99e5cf6fb8ff515b4 2023-04-19 07:54:54,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/69576c2ad1df4f36beb19e30348da8b8 2023-04-19 07:54:54,075 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/12f00c3c56c94c1e984cb229f8e34f6e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/12f00c3c56c94c1e984cb229f8e34f6e 2023-04-19 07:54:54,080 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/12f00c3c56c94c1e984cb229f8e34f6e, entries=200, sequenceid=342, filesize=14.3 K 2023-04-19 07:54:54,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/7b88c50217604ea99e5cf6fb8ff515b4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7b88c50217604ea99e5cf6fb8ff515b4 2023-04-19 07:54:54,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47274 deadline: 1681890954078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,084 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7b88c50217604ea99e5cf6fb8ff515b4, entries=150, sequenceid=342, filesize=11.9 K 2023-04-19 07:54:54,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/69576c2ad1df4f36beb19e30348da8b8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/69576c2ad1df4f36beb19e30348da8b8 2023-04-19 07:54:54,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890954080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,094 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:54,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:54,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:54,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/69576c2ad1df4f36beb19e30348da8b8, entries=150, sequenceid=342, filesize=11.9 K 2023-04-19 07:54:54,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:54,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:54,098 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:54,098 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:54,098 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:54,098 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:54,098 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,098 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,099 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for cfaed3abed1fbb0d5a48467b790cb91c in 101ms, sequenceid=342, compaction requested=true 2023-04-19 07:54:54,099 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:54,099 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:54,099 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:54:54,099 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:54,099 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:54:54,099 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:54,100 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:54,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:54,100 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,100 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,100 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:54,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:54,101 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58286 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:54:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,101 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/B is initiating minor compaction (all files) 2023-04-19 07:54:54,101 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60726 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:54:54,101 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/B in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,101 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/A is initiating minor compaction (all files) 2023-04-19 07:54:54,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:54,102 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7128a63e249b4b4192e8b679e4c9a146, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/66841a259cd842db8ef92f44cf587df7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/9edbb119ada84340ba2e4f7824b91381, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/f20233317cda4a4c880978d72831f676, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7b88c50217604ea99e5cf6fb8ff515b4] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=56.9 K 2023-04-19 07:54:54,102 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/A in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,102 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/990964db056a47ebbfdf93750f2bda4d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/6b8487b5f6614f8fb38de11a9d6891a9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/1d5ad41481224c9a84c2b0e91b0058bf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/2976b925835d494dacfbd460bfa387f4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/12f00c3c56c94c1e984cb229f8e34f6e] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=59.3 K 2023-04-19 07:54:54,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:54,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:54,102 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:54,102 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:54,102 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:54,102 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7128a63e249b4b4192e8b679e4c9a146, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681890890296 2023-04-19 07:54:54,102 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 990964db056a47ebbfdf93750f2bda4d, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681890890296 2023-04-19 07:54:54,103 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 66841a259cd842db8ef92f44cf587df7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681890890787 2023-04-19 07:54:54,103 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 6b8487b5f6614f8fb38de11a9d6891a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681890890787 2023-04-19 07:54:54,103 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 9edbb119ada84340ba2e4f7824b91381, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681890890808 2023-04-19 07:54:54,103 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 1d5ad41481224c9a84c2b0e91b0058bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681890890808 2023-04-19 07:54:54,104 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 2976b925835d494dacfbd460bfa387f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681890891856 2023-04-19 07:54:54,104 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:54,104 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:54,104 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting f20233317cda4a4c880978d72831f676, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681890891856 2023-04-19 07:54:54,104 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,104 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:54,104 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-19 07:54:54,104 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 12f00c3c56c94c1e984cb229f8e34f6e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681890893993 2023-04-19 07:54:54,105 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:54:54,105 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7b88c50217604ea99e5cf6fb8ff515b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681890893993 2023-04-19 07:54:54,105 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:54,105 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:54:54,105 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:54,105 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:54:54,105 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:54,117 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#B#compaction#208 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:54,126 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#A#compaction#209 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:54,152 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/e24d5bf9de9640caaa8c2fabe1053aa1 2023-04-19 07:54:54,175 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/e73f89c1828d4cc6b46a1f98346f0bc2 2023-04-19 07:54:54,180 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/d5911e040aa742b49793db1dbca9ef29 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/d5911e040aa742b49793db1dbca9ef29 2023-04-19 07:54:54,180 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/4bb93a4c6998434b821785ae9b8fe3e2 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/4bb93a4c6998434b821785ae9b8fe3e2 2023-04-19 07:54:54,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:54,187 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:54,189 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/B of cfaed3abed1fbb0d5a48467b790cb91c into d5911e040aa742b49793db1dbca9ef29(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:54,189 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:54,189 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/B, priority=11, startTime=1681890894099; duration=0sec 2023-04-19 07:54:54,189 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:54,189 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:54:54,191 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58286 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:54:54,191 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/C is initiating minor compaction (all files) 2023-04-19 07:54:54,191 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/C in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,191 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/51bf9d43fbe84353b20a06297fd7105d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/dff4b51069a4486ab11f8d3aa6cd9411, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cf79064f7b364df8838a7c129ff79e96, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/c42cac8819944f3b925e047944c9fc34, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/69576c2ad1df4f36beb19e30348da8b8] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=56.9 K 2023-04-19 07:54:54,191 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 51bf9d43fbe84353b20a06297fd7105d, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681890890296 2023-04-19 07:54:54,192 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/bbab1ffb7e0446cbb067c8cc54d5cf0f 2023-04-19 07:54:54,192 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting dff4b51069a4486ab11f8d3aa6cd9411, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1681890890787 2023-04-19 07:54:54,193 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting cf79064f7b364df8838a7c129ff79e96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1681890890808 2023-04-19 07:54:54,193 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting c42cac8819944f3b925e047944c9fc34, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1681890891856 2023-04-19 07:54:54,193 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 69576c2ad1df4f36beb19e30348da8b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681890893993 2023-04-19 07:54:54,199 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/e24d5bf9de9640caaa8c2fabe1053aa1 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e24d5bf9de9640caaa8c2fabe1053aa1 2023-04-19 07:54:54,199 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/A of cfaed3abed1fbb0d5a48467b790cb91c into 4bb93a4c6998434b821785ae9b8fe3e2(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:54,199 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:54,199 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/A, priority=11, startTime=1681890894099; duration=0sec 2023-04-19 07:54:54,199 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:54,204 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#C#compaction#212 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:54,204 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e24d5bf9de9640caaa8c2fabe1053aa1, entries=150, sequenceid=378, filesize=11.9 K 2023-04-19 07:54:54,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47274 deadline: 1681890954200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,208 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/e73f89c1828d4cc6b46a1f98346f0bc2 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/e73f89c1828d4cc6b46a1f98346f0bc2 2023-04-19 07:54:54,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890954204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,212 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/e73f89c1828d4cc6b46a1f98346f0bc2, entries=150, sequenceid=378, filesize=11.9 K 2023-04-19 07:54:54,213 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/bbab1ffb7e0446cbb067c8cc54d5cf0f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/bbab1ffb7e0446cbb067c8cc54d5cf0f 2023-04-19 07:54:54,217 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/bbab1ffb7e0446cbb067c8cc54d5cf0f, entries=150, sequenceid=378, filesize=11.9 K 2023-04-19 07:54:54,218 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for cfaed3abed1fbb0d5a48467b790cb91c in 114ms, sequenceid=378, compaction requested=false 2023-04-19 07:54:54,218 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:54,218 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:54,222 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,222 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,223 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,223 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:54,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28920d2e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:54,225 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:54,225 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,226 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,226 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:54,226 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:54,226 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:54,227 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,227 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:54,229 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:54,229 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:54,229 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:54,229 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:54,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:54,233 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:54,233 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:54,233 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:54,233 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:54,233 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:54,235 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,235 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,235 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:54,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:54,236 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,238 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,241 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,241 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,241 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,241 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:54,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:54,241 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:54,241 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:54,241 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:54,241 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,242 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:54,241 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:54,241 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:54,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:54,242 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:54,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:54,307 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 07:54:54,307 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:54:54,307 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:54,308 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:54:54,308 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:54,308 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:54:54,308 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:54,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/c23ad1d32f3247a894b49cd5336a94c0 2023-04-19 07:54:54,342 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:54,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:54,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:54,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:54,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:54,346 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:54,346 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:54,346 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:54,346 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:54,347 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,347 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:54,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:54,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:54,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:54,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:54,350 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,350 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:54,350 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:54,350 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:54,351 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:54,355 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,355 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,356 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,356 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:54,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61968e1b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:54,357 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:54,357 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,358 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,358 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:54,358 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:54,359 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:54,359 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,359 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:54,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890894350, max timestamp=1681890894357Num uniques 859; 2023-04-19 07:54:54,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:54,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890894350, max timestamp=1681890894357Num uniques 859; 2023-04-19 07:54:54,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:54,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681890894350, max timestamp=1681890894357Num uniques 859; 2023-04-19 07:54:54,360 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:54,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-19 07:54:54,360 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:54,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:54,363 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:54,363 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:54,363 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:54,363 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:54,363 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:54,369 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,369 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,369 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:54,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,371 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,372 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,376 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,376 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,376 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:54,376 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,376 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:54,376 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:54,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:54,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:54,376 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:54,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:54,376 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,377 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:54,377 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:54,377 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,377 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:54,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890954385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890954386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,477 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:54,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:54,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:54,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:54,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:54,481 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:54,481 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:54,481 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:54,481 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:54,481 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,482 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,483 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:54,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:54,483 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:54,483 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,483 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:54,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:54,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:54,484 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:54,485 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,485 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:54,485 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:54,485 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:54,487 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,487 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,487 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,487 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:54,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b3d1ab6[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:54,489 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:54,489 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,490 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:54,490 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:54,490 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,490 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:54,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890954488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890954490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,493 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:54,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:54,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:54,495 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:54,495 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:54,496 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:54,496 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:54,496 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:54,497 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,497 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,498 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,498 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:54,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,500 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,504 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:54,504 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:54,504 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:54,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:54,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:54,504 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,504 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,505 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,505 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,504 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:54,505 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:54,505 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:54,505 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,505 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:54,605 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:54,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:54,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:54,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:54,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:54,609 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:54,609 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:54,609 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:54,609 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:54,609 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,609 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,612 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:54,612 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:54,612 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,612 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:54,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:54,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:54,614 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,614 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:54,614 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:54,614 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:54,614 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:54,616 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,616 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:54,618 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:54,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15824be8[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:54,618 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,619 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,619 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:54,619 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:54,621 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:54,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:54,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:54,624 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:54,624 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:54,624 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:54,624 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:54,624 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:54,627 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,627 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,627 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,627 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/7e22d0ab9d9d4916928b9c427b277453 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/7e22d0ab9d9d4916928b9c427b277453 2023-04-19 07:54:54,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:54,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,632 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,632 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/C of cfaed3abed1fbb0d5a48467b790cb91c into 7e22d0ab9d9d4916928b9c427b277453(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:54,632 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:54,632 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/C, priority=11, startTime=1681890894099; duration=0sec 2023-04-19 07:54:54,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,632 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,633 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:54,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,635 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:54,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:54,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:54,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:54,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:54,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:54,636 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:54,636 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:54,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:54,636 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,636 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:54,636 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,636 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890954692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890954693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,736 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:54,737 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/33032b91ab9546718acc1fc760c57984 2023-04-19 07:54:54,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:54,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:54,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:54,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:54,741 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:54,741 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:54,741 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:54,741 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:54,742 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,742 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,744 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:54,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:54,744 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,744 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,744 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:54,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:54,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:54,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:54,748 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,748 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:54,748 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:54,748 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:54,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,750 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,750 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/31a3b439d6624363bef9f771bada13bc 2023-04-19 07:54:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,751 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,751 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:54,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d41cf0b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:54,753 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:54,753 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,755 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/c23ad1d32f3247a894b49cd5336a94c0 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/c23ad1d32f3247a894b49cd5336a94c0 2023-04-19 07:54:54,756 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,756 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:54,756 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:54,756 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:54,756 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,756 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:54,757 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,757 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:54,757 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,759 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/c23ad1d32f3247a894b49cd5336a94c0, entries=150, sequenceid=391, filesize=11.9 K 2023-04-19 07:54:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,760 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/33032b91ab9546718acc1fc760c57984 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/33032b91ab9546718acc1fc760c57984 2023-04-19 07:54:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:54,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:54,761 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:54,761 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:54,761 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:54,761 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:54,761 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:54,763 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,763 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:54,763 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:54,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:54,763 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:54,765 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/33032b91ab9546718acc1fc760c57984, entries=150, sequenceid=391, filesize=11.9 K 2023-04-19 07:54:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:54,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/31a3b439d6624363bef9f771bada13bc as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/31a3b439d6624363bef9f771bada13bc 2023-04-19 07:54:54,766 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,768 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,768 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:54,768 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,768 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:54,768 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:54,768 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:54,768 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:54,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:54,769 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,769 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:54,769 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:54,769 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:54,769 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:54,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:54,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/31a3b439d6624363bef9f771bada13bc, entries=150, sequenceid=391, filesize=11.9 K 2023-04-19 07:54:54,771 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.08 KB/130125 for cfaed3abed1fbb0d5a48467b790cb91c in 464ms, sequenceid=391, compaction requested=true 2023-04-19 07:54:54,771 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:54,771 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:54,771 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:54,771 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:54:54,771 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:54:54,771 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:54,772 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36730 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:54:54,772 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/B is initiating minor compaction (all files) 2023-04-19 07:54:54,772 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36730 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:54:54,772 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/B in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,772 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/A is initiating minor compaction (all files) 2023-04-19 07:54:54,772 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/A in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,772 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/d5911e040aa742b49793db1dbca9ef29, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/e73f89c1828d4cc6b46a1f98346f0bc2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/33032b91ab9546718acc1fc760c57984] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=35.9 K 2023-04-19 07:54:54,772 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/4bb93a4c6998434b821785ae9b8fe3e2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e24d5bf9de9640caaa8c2fabe1053aa1, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/c23ad1d32f3247a894b49cd5336a94c0] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=35.9 K 2023-04-19 07:54:54,773 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting d5911e040aa742b49793db1dbca9ef29, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681890893993 2023-04-19 07:54:54,773 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 4bb93a4c6998434b821785ae9b8fe3e2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681890893993 2023-04-19 07:54:54,773 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting e73f89c1828d4cc6b46a1f98346f0bc2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681890894074 2023-04-19 07:54:54,774 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting e24d5bf9de9640caaa8c2fabe1053aa1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681890894074 2023-04-19 07:54:54,774 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 33032b91ab9546718acc1fc760c57984, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681890894195 2023-04-19 07:54:54,774 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting c23ad1d32f3247a894b49cd5336a94c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681890894195 2023-04-19 07:54:54,785 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#B#compaction#216 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:54,787 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#A#compaction#217 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:54,869 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:54,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:54,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:54,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:54,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:54,873 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:54,873 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:54,873 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:54,873 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:54,873 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,873 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,876 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:54,876 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,877 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,877 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:54,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:54,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:54,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:54,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:54,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:54,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:54,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:54,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:54,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:54,878 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:54,878 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:54,878 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=127.08 KB heapSize=364.15 KB 2023-04-19 07:54:54,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:54,879 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:54,879 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:54:54,879 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:54,879 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:54:54,879 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:54,879 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:54:54,879 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:54,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890954902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47286 deadline: 1681890954960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,965 DEBUG [Thread-1533] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:54,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47236 deadline: 1681890954970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,974 DEBUG [Thread-1535] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:54,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890954995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:54,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:54,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890954995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:55,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890955006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,204 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/cb5f0bfd751c413486a64fe441cb33db as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/cb5f0bfd751c413486a64fe441cb33db 2023-04-19 07:54:55,208 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/B of cfaed3abed1fbb0d5a48467b790cb91c into cb5f0bfd751c413486a64fe441cb33db(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:55,208 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:55,208 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/B, priority=13, startTime=1681890894771; duration=0sec 2023-04-19 07:54:55,209 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:55,209 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:54:55,210 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36730 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:54:55,210 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/C is initiating minor compaction (all files) 2023-04-19 07:54:55,210 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/C in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:55,210 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/7e22d0ab9d9d4916928b9c427b277453, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/bbab1ffb7e0446cbb067c8cc54d5cf0f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/31a3b439d6624363bef9f771bada13bc] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=35.9 K 2023-04-19 07:54:55,211 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7e22d0ab9d9d4916928b9c427b277453, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681890893993 2023-04-19 07:54:55,211 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting bbab1ffb7e0446cbb067c8cc54d5cf0f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681890894074 2023-04-19 07:54:55,212 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/13ff2cc14f434265afdb8036286726bc as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/13ff2cc14f434265afdb8036286726bc 2023-04-19 07:54:55,212 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 31a3b439d6624363bef9f771bada13bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681890894195 2023-04-19 07:54:55,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:55,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890955209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,222 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/A of cfaed3abed1fbb0d5a48467b790cb91c into 13ff2cc14f434265afdb8036286726bc(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:55,222 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:55,222 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/A, priority=13, startTime=1681890894771; duration=0sec 2023-04-19 07:54:55,222 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:55,223 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#C#compaction#219 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:55,298 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/eadc4b6a18a24821b00e405d13c25bf5 2023-04-19 07:54:55,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:55,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890955498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:55,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890955500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:55,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890955513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,639 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/14d4899ba4204510b09f1193b4a3ace9 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/14d4899ba4204510b09f1193b4a3ace9 2023-04-19 07:54:55,644 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/C of cfaed3abed1fbb0d5a48467b790cb91c into 14d4899ba4204510b09f1193b4a3ace9(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:55,644 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:55,644 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/C, priority=13, startTime=1681890894771; duration=0sec 2023-04-19 07:54:55,644 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:55,709 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/5c81d91cc8fb436ea50aa7d3ff737219 2023-04-19 07:54:55,755 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/5a69ac8b56464f62bccca8fc1fc4cc8d 2023-04-19 07:54:55,759 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/eadc4b6a18a24821b00e405d13c25bf5 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/eadc4b6a18a24821b00e405d13c25bf5 2023-04-19 07:54:55,764 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/eadc4b6a18a24821b00e405d13c25bf5, entries=200, sequenceid=429, filesize=14.3 K 2023-04-19 07:54:55,765 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/5c81d91cc8fb436ea50aa7d3ff737219 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/5c81d91cc8fb436ea50aa7d3ff737219 2023-04-19 07:54:55,769 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/5c81d91cc8fb436ea50aa7d3ff737219, entries=150, sequenceid=429, filesize=11.9 K 2023-04-19 07:54:55,770 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/5a69ac8b56464f62bccca8fc1fc4cc8d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/5a69ac8b56464f62bccca8fc1fc4cc8d 2023-04-19 07:54:55,774 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/5a69ac8b56464f62bccca8fc1fc4cc8d, entries=150, sequenceid=429, filesize=11.9 K 2023-04-19 07:54:55,775 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~133.78 KB/136995, heapSize ~384.02 KB/393240, currentSize=46.96 KB/48090 for cfaed3abed1fbb0d5a48467b790cb91c in 897ms, sequenceid=429, compaction requested=false 2023-04-19 07:54:55,775 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:55,775 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:55,775 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:55,775 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:55,775 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,775 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:55,775 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:55,777 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,777 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:55,777 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:55,777 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:55,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:55,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78727aee[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:55,779 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:55,779 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:55,780 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:55,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:55,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:55,780 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,780 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:55,783 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:55,783 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,783 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:55,783 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:55,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:55,786 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:55,786 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:55,786 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:55,786 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:55,786 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:55,788 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:55,788 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:55,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:55,788 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:55,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:55,789 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:55,791 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,796 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,796 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:55,796 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:55,796 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:55,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:55,796 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:55,797 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,797 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:55,797 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:55,797 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:55,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:55,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:55,797 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:55,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:55,797 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:55,797 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:55,897 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:55,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:55,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:55,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:55,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:55,901 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:55,901 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:55,901 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:55,901 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:55,902 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:55,902 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:55,904 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,904 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:55,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:55,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:55,904 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:55,904 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:55,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:55,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:55,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:55,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:55,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:55,906 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:55,906 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:55,906 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:55,906 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:55,906 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:55,906 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:55,906 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:55,907 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:54:55,907 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:54:55,908 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:55,908 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:54:55,908 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:55,908 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:54:55,908 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:55,926 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/974cf33b9a454130909ebe1991a97a96 2023-04-19 07:54:56,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:56,019 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:56,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890896077, max timestamp=1681890896090Num uniques 859; 2023-04-19 07:54:56,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:56,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890896077, max timestamp=1681890896090Num uniques 859; 2023-04-19 07:54:56,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890896077, max timestamp=1681890896090Num uniques 859; 2023-04-19 07:54:56,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:56,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:56,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:56,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890956157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:56,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890956261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,359 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/7078f71258fd437aa8f1fdfd61f18b50 2023-04-19 07:54:56,378 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/147aec96eea244509d724a735b1cd74b 2023-04-19 07:54:56,383 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/974cf33b9a454130909ebe1991a97a96 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/974cf33b9a454130909ebe1991a97a96 2023-04-19 07:54:56,389 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/974cf33b9a454130909ebe1991a97a96, entries=150, sequenceid=442, filesize=11.9 K 2023-04-19 07:54:56,390 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/7078f71258fd437aa8f1fdfd61f18b50 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7078f71258fd437aa8f1fdfd61f18b50 2023-04-19 07:54:56,394 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7078f71258fd437aa8f1fdfd61f18b50, entries=150, sequenceid=442, filesize=11.9 K 2023-04-19 07:54:56,395 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/147aec96eea244509d724a735b1cd74b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/147aec96eea244509d724a735b1cd74b 2023-04-19 07:54:56,400 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/147aec96eea244509d724a735b1cd74b, entries=150, sequenceid=442, filesize=11.9 K 2023-04-19 07:54:56,401 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for cfaed3abed1fbb0d5a48467b790cb91c in 494ms, sequenceid=442, compaction requested=true 2023-04-19 07:54:56,401 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:56,401 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:56,401 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:56,401 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:56,401 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,401 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:56,401 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:56,404 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:56,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b9927fc[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:56,406 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:56,406 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,407 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,407 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:56,407 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:56,407 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:56,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,408 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:56,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,409 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:56,409 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:56,409 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:56,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:56,412 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:56,412 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:56,412 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:56,412 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:56,412 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:56,413 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,413 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,414 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,415 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,417 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,420 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:56,420 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:56,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:56,421 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,421 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,421 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,421 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:56,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:56,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:56,421 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:56,421 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,421 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,421 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,421 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:56,421 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:56,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:56,468 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-19 07:54:56,469 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:54:56,469 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 07:54:56,469 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:54:56,469 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 07:54:56,469 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:54:56,469 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 07:54:56,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/74b5f1bda1244e7ba48ca2486be938e6 2023-04-19 07:54:56,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:56,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890956504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:56,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890956505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:56,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890956505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,521 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:56,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:56,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:56,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:56,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:56,525 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:56,525 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:56,525 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:56,525 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:56,525 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,525 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,526 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:56,527 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,527 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,527 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:56,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:56,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:56,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:56,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:56,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:56,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:56,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:56,529 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:56,529 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:56,529 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:56,529 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:56,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:56,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:56,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:56,529 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:56,531 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,531 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,531 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,531 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:56,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c43ee6c[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:56,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:56,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,534 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:56,535 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,535 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:56,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:56,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:56,537 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,537 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,537 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:56,537 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:56,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:56,540 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:56,540 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:56,540 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:56,540 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:56,540 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:56,542 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,542 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,542 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:56,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,543 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,544 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,546 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:56,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:56,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:56,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:56,547 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:56,547 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,547 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,547 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,547 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,547 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,547 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:56,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:56,548 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:56,548 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:56,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:56,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:56,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890956608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:56,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890956610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,648 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:56,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:56,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:56,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:56,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:56,651 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:56,651 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:56,651 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:56,651 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:56,652 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,652 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,653 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:56,653 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:56,653 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,653 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:56,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:56,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:56,655 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:56,655 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:56,655 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:56,655 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:56,655 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:56,655 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:56,655 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:56,655 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:56,655 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:56,655 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:56,655 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:56,655 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,655 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:56,655 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:56,657 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,657 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,658 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,658 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:56,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:56,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f595f78[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:56,660 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:56,660 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,661 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,661 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:56,661 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:56,661 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:56,661 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,661 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:56,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:56,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:56,663 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:56,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:56,665 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:56,665 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:56,665 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:56,665 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:56,665 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:56,667 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,667 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,667 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:56,668 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,669 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:56,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,673 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:56,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:56,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:56,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:56,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:56,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:56,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:56,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,673 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:56,673 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:56,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:56,773 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:56,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:56,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:56,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:56,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:56,777 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:56,777 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:56,777 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:56,777 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:56,777 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,777 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,784 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,784 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:56,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:56,785 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,785 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:56,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:56,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:56,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:56,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:56,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:56,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:56,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:56,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:56,786 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:56,786 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:56,787 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:56,787 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:56,787 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:56,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,788 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:56,790 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:56,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3497eb77[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:56,790 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,792 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,792 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:56,792 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:56,793 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:56,793 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,793 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:56,794 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:56,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:56,797 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:56,797 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:56,797 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:56,797 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:56,797 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:56,798 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,798 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,798 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:56,799 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,800 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,802 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,802 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,802 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,802 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,802 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:56,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:56,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:56,802 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,802 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,802 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,802 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:56,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:56,802 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:56,802 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:56,803 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:56,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:56,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:56,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:56,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890956810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:56,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47214 deadline: 1681890956813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,903 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:56,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:56,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:56,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:56,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:56,906 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:56,906 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:56,906 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:56,906 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:56,906 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,906 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:56,908 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,909 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,909 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:56,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:56,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:56,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:56,910 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:56,910 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:56,911 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:56,911 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:56,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/53916dfb4b454864bf96cfdce5e1d68c 2023-04-19 07:54:56,912 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,912 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,913 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,913 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:56,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6039cde[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:56,914 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:56,914 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,915 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:56,916 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,916 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:56,917 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,917 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:56,917 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:56,917 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:56,921 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:56,921 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:56,921 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:56,921 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:56,921 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:56,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/dbabd0a9ab4c4c9f97340f34806e01d4 2023-04-19 07:54:56,923 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:56,923 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,924 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:56,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:56,924 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:56,925 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:56,927 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/74b5f1bda1244e7ba48ca2486be938e6 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/74b5f1bda1244e7ba48ca2486be938e6 2023-04-19 07:54:56,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:56,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:56,928 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:56,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:56,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:56,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:56,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:56,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:56,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:56,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:56,928 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:56,928 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:56,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:56,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:56,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:56,931 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/74b5f1bda1244e7ba48ca2486be938e6, entries=200, sequenceid=480, filesize=14.3 K 2023-04-19 07:54:56,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/53916dfb4b454864bf96cfdce5e1d68c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/53916dfb4b454864bf96cfdce5e1d68c 2023-04-19 07:54:56,935 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/53916dfb4b454864bf96cfdce5e1d68c, entries=150, sequenceid=480, filesize=11.9 K 2023-04-19 07:54:56,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/dbabd0a9ab4c4c9f97340f34806e01d4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/dbabd0a9ab4c4c9f97340f34806e01d4 2023-04-19 07:54:56,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/dbabd0a9ab4c4c9f97340f34806e01d4, entries=150, sequenceid=480, filesize=11.9 K 2023-04-19 07:54:56,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for cfaed3abed1fbb0d5a48467b790cb91c in 472ms, sequenceid=480, compaction requested=true 2023-04-19 07:54:56,940 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:56,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:56,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:56,940 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:56,940 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:56,940 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:56,942 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49018 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:56,942 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/B is initiating minor compaction (all files) 2023-04-19 07:54:56,942 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/B in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:56,942 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/cb5f0bfd751c413486a64fe441cb33db, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/5c81d91cc8fb436ea50aa7d3ff737219, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7078f71258fd437aa8f1fdfd61f18b50, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/53916dfb4b454864bf96cfdce5e1d68c] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=47.9 K 2023-04-19 07:54:56,942 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53898 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:56,942 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/A is initiating minor compaction (all files) 2023-04-19 07:54:56,942 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/A in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:56,942 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/13ff2cc14f434265afdb8036286726bc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/eadc4b6a18a24821b00e405d13c25bf5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/974cf33b9a454130909ebe1991a97a96, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/74b5f1bda1244e7ba48ca2486be938e6] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=52.6 K 2023-04-19 07:54:56,944 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 13ff2cc14f434265afdb8036286726bc, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681890894195 2023-04-19 07:54:56,944 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting cb5f0bfd751c413486a64fe441cb33db, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681890894195 2023-04-19 07:54:56,944 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting eadc4b6a18a24821b00e405d13c25bf5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681890894382 2023-04-19 07:54:56,944 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 5c81d91cc8fb436ea50aa7d3ff737219, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681890894382 2023-04-19 07:54:56,944 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 974cf33b9a454130909ebe1991a97a96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1681890894888 2023-04-19 07:54:56,944 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7078f71258fd437aa8f1fdfd61f18b50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1681890894888 2023-04-19 07:54:56,945 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 74b5f1bda1244e7ba48ca2486be938e6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681890896109 2023-04-19 07:54:56,945 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 53916dfb4b454864bf96cfdce5e1d68c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681890896109 2023-04-19 07:54:56,954 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#A#compaction#229 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:56,954 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#B#compaction#228 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:57,028 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:57,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:57,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:57,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:57,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:57,032 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:57,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:57,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:57,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:57,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:57,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:57,033 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:57,033 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:57,034 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,034 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:57,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:57,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:57,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:57,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:57,035 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:57,035 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:57,035 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:57,035 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:57,035 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:57,035 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:57,035 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:57,035 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 07:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:57,043 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/df7a62efa0914a7bb9b2cf724209b3b8 2023-04-19 07:54:57,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:57,117 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:57,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890897137, max timestamp=1681890897144Num uniques 859; 2023-04-19 07:54:57,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:57,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890897137, max timestamp=1681890897144Num uniques 859; 2023-04-19 07:54:57,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890897137, max timestamp=1681890897144Num uniques 859; 2023-04-19 07:54:57,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:57,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:57,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:57,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890957176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890897166, max timestamp=1681890897178Num uniques 1018; 2023-04-19 07:54:57,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:54:57,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890897166, max timestamp=1681890897178Num uniques 1018; 2023-04-19 07:54:57,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:54:57,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890897166, max timestamp=1681890897178Num uniques 1018; 2023-04-19 07:54:57,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:54:57,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681890897229, max timestamp=1681890897240Num uniques 1027; 2023-04-19 07:54:57,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681890897229, max timestamp=1681890897240Num uniques 1027; 2023-04-19 07:54:57,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-19 07:54:57,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681890897229, max timestamp=1681890897240Num uniques 1027; 2023-04-19 07:54:57,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-19 07:54:57,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-19 07:54:57,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681890897279, max timestamp=1681890897291Num uniques 1036; 2023-04-19 07:54:57,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-19 07:54:57,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681890897279, max timestamp=1681890897291Num uniques 1036; 2023-04-19 07:54:57,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-19 07:54:57,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681890897279, max timestamp=1681890897291Num uniques 1036; 2023-04-19 07:54:57,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-19 07:54:57,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:57,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890957347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:57,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890957351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,365 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/088d6a69fb34477aab615c8f16ebeb26 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/088d6a69fb34477aab615c8f16ebeb26 2023-04-19 07:54:57,377 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/796faeb88eab4e89bc495244a23d35fc as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/796faeb88eab4e89bc495244a23d35fc 2023-04-19 07:54:57,383 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/A of cfaed3abed1fbb0d5a48467b790cb91c into 088d6a69fb34477aab615c8f16ebeb26(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:57,384 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:57,384 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/A, priority=12, startTime=1681890896940; duration=0sec 2023-04-19 07:54:57,384 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:57,384 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:57,384 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/B of cfaed3abed1fbb0d5a48467b790cb91c into 796faeb88eab4e89bc495244a23d35fc(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:57,384 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:57,384 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/B, priority=12, startTime=1681890896940; duration=0sec 2023-04-19 07:54:57,384 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:57,386 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49018 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:57,386 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/C is initiating minor compaction (all files) 2023-04-19 07:54:57,386 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/C in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:57,386 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/14d4899ba4204510b09f1193b4a3ace9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/5a69ac8b56464f62bccca8fc1fc4cc8d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/147aec96eea244509d724a735b1cd74b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/dbabd0a9ab4c4c9f97340f34806e01d4] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=47.9 K 2023-04-19 07:54:57,386 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 14d4899ba4204510b09f1193b4a3ace9, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681890894195 2023-04-19 07:54:57,387 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 5a69ac8b56464f62bccca8fc1fc4cc8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1681890894382 2023-04-19 07:54:57,387 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 147aec96eea244509d724a735b1cd74b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1681890894888 2023-04-19 07:54:57,387 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting dbabd0a9ab4c4c9f97340f34806e01d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681890896109 2023-04-19 07:54:57,395 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#C#compaction#232 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:57,407 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/ce3fbc0fc9b24a418cecc7d69a6ba164 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/ce3fbc0fc9b24a418cecc7d69a6ba164 2023-04-19 07:54:57,412 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/C of cfaed3abed1fbb0d5a48467b790cb91c into ce3fbc0fc9b24a418cecc7d69a6ba164(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:57,412 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:57,412 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/C, priority=12, startTime=1681890896940; duration=0sec 2023-04-19 07:54:57,412 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:57,454 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/55df49fc0dd04232aa4dc8497ee9d80d 2023-04-19 07:54:57,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:57,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890957452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:57,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890957455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,469 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/63f2c010f22947ac891f8876aac75630 2023-04-19 07:54:57,476 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/df7a62efa0914a7bb9b2cf724209b3b8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/df7a62efa0914a7bb9b2cf724209b3b8 2023-04-19 07:54:57,481 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/df7a62efa0914a7bb9b2cf724209b3b8, entries=150, sequenceid=489, filesize=11.9 K 2023-04-19 07:54:57,481 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/55df49fc0dd04232aa4dc8497ee9d80d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/55df49fc0dd04232aa4dc8497ee9d80d 2023-04-19 07:54:57,486 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/55df49fc0dd04232aa4dc8497ee9d80d, entries=150, sequenceid=489, filesize=11.9 K 2023-04-19 07:54:57,487 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/63f2c010f22947ac891f8876aac75630 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/63f2c010f22947ac891f8876aac75630 2023-04-19 07:54:57,490 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/63f2c010f22947ac891f8876aac75630, entries=150, sequenceid=489, filesize=11.9 K 2023-04-19 07:54:57,492 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for cfaed3abed1fbb0d5a48467b790cb91c in 457ms, sequenceid=489, compaction requested=false 2023-04-19 07:54:57,492 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:57,492 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:57,492 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:57,492 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:57,492 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:57,494 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65d032fc[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:57,496 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:57,497 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:57,498 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:57,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:57,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:57,499 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:57,499 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:57,499 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,499 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:57,500 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:57,500 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:57,500 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:57,500 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:57,503 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:57,503 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:57,503 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:57,503 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:57,503 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:57,505 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:57,505 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:57,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:57,505 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:57,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:57,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,508 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,512 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:57,512 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:57,512 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:57,512 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:57,512 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:57,512 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:57,512 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:57,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:57,512 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:57,513 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:57,513 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:57,513 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:57,513 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:57,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:57,613 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:57,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:57,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:57,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:57,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:57,616 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:57,617 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:57,617 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:57,617 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:57,617 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:57,617 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:57,619 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:57,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:57,619 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,619 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:57,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:57,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:57,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:57,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:57,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:57,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:57,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:57,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:57,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:57,621 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:57,621 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:57,621 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=140.89 KB heapSize=397.69 KB 2023-04-19 07:54:57,621 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:54:57,621 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:57,621 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:54:57,622 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:57,622 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:54:57,622 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:57,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:57,660 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:57,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:57,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890957667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:57,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890957669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:57,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890957770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:57,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890957773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:57,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890957974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:57,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:57,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890957976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,030 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/ef1ecbcf894d460c9b049d1ba1e20e84 2023-04-19 07:54:58,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:58,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890958277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:58,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890958279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,440 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/50c0bba6e7834f979810d3c7aab148d4 2023-04-19 07:54:58,451 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/6f12d53494de412c9fcd34096d9213e3 2023-04-19 07:54:58,455 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/ef1ecbcf894d460c9b049d1ba1e20e84 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/ef1ecbcf894d460c9b049d1ba1e20e84 2023-04-19 07:54:58,459 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/ef1ecbcf894d460c9b049d1ba1e20e84, entries=150, sequenceid=580, filesize=11.9 K 2023-04-19 07:54:58,459 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-19 07:54:58,460 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/50c0bba6e7834f979810d3c7aab148d4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/50c0bba6e7834f979810d3c7aab148d4 2023-04-19 07:54:58,464 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/50c0bba6e7834f979810d3c7aab148d4, entries=150, sequenceid=580, filesize=11.9 K 2023-04-19 07:54:58,465 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/6f12d53494de412c9fcd34096d9213e3 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/6f12d53494de412c9fcd34096d9213e3 2023-04-19 07:54:58,472 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/6f12d53494de412c9fcd34096d9213e3, entries=150, sequenceid=580, filesize=11.9 K 2023-04-19 07:54:58,473 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~397.64 KB/407184, currentSize=40.25 KB/41220 for cfaed3abed1fbb0d5a48467b790cb91c in 852ms, sequenceid=580, compaction requested=true 2023-04-19 07:54:58,473 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:58,473 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:58,473 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:58,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:58,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:58,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:58,486 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,486 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:58,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:58,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@706703a1[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:58,488 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:58,489 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,497 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:58,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:58,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:58,498 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,498 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:58,500 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,500 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:58,500 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:58,500 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:58,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:58,503 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:58,503 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:58,503 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:58,503 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:58,503 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:58,505 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,505 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:58,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,505 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:58,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:58,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,507 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,510 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,510 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,510 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,510 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:58,510 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:58,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:58,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:58,510 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:58,510 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:58,510 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:58,510 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,511 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,511 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:58,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:58,511 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:58,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:58,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:58,522 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:54:58,522 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:54:58,522 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:58,522 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:54:58,522 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:58,522 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:54:58,522 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:54:58,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890898564, max timestamp=1681890898583Num uniques 859; 2023-04-19 07:54:58,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:58,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890898564, max timestamp=1681890898583Num uniques 859; 2023-04-19 07:54:58,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:58,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890898564, max timestamp=1681890898583Num uniques 859; 2023-04-19 07:54:58,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:54:58,611 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:58,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:58,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:58,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:58,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:58,614 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:58,614 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:58,614 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:58,614 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:58,615 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,615 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,616 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:58,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:58,616 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:58,617 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,617 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:58,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:58,618 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:58,618 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:58,618 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:58,618 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:58,618 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:58,618 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:58,618 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:58,618 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:58,618 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:58,618 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:58,618 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:58,618 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:58,619 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:58,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,621 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:58,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:58,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:58,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ba09289[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:58,623 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:58,623 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,624 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,624 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:58,624 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:58,624 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:58,624 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,624 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:58,626 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,626 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:58,626 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:58,626 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:58,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:58,628 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:58,628 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:58,628 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:58,628 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:58,628 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:58,630 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,630 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:58,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,630 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:58,630 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:58,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,631 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:58,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:58,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:58,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:58,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:58,635 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:58,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:58,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,635 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:58,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,635 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:58,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:58,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:58,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890958637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,735 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:58,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:58,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:58,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:58,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:58,739 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:58,739 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:58,739 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:58,739 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:58,739 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,739 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,740 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:58,741 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:58,741 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,741 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:58,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:58,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:58,742 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:58,742 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:58,742 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:58,742 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:58,742 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:58,742 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:58,743 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:58,743 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:58,743 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:58,743 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:58,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:58,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890958740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,744 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,744 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:58,745 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,745 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:58,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:58,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61df144f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:58,746 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:58,746 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,747 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,748 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:58,748 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:58,748 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:58,748 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,748 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:58,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:58,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:58,752 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:58,752 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:58,752 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:58,752 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:58,752 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:58,754 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,754 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,754 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:58,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:58,754 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,755 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,758 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,758 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,758 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,758 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,758 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:58,758 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:58,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:58,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:58,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:58,758 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:58,758 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:58,758 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,758 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:58,758 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:58,758 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:58,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:58,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890958779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:58,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890958782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,858 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:58,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:58,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:58,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:58,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:58,862 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:58,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:58,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:58,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:58,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,863 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,863 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:58,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:58,864 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,864 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:58,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:58,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:58,865 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:58,866 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:58,866 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:58,866 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:58,866 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:58,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:58,869 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,869 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:58,869 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,869 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:58,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:58,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:58,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:58,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57c9d5a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:58,871 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:58,871 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,872 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:58,872 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:58,872 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:58,872 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,872 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:58,876 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,876 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:58,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:58,878 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:58,878 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:58,878 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:58,878 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:58,878 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:58,881 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,881 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:58,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,881 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:58,881 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:58,883 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,885 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,885 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,885 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:58,885 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:58,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:58,885 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,885 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,885 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,885 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,886 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:58,886 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:58,886 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:58,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:58,886 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:58,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:58,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:58,932 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/4660cf6cc6924f8fb11ce032ca34676d 2023-04-19 07:54:58,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:58,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890958944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:58,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47286 deadline: 1681890958971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,974 DEBUG [Thread-1533] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:58,986 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:58,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:58,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:58,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:58,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:58,989 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:58,989 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:58,989 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:58,989 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:58,989 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,989 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,991 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,991 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:58,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:58,991 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,991 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:58,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:58,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:58,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:58,992 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:58,992 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:58,992 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:58,992 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:58,992 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:58,993 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:58,993 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:58,993 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:58,993 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:58,993 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:58,994 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,994 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:58,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:58,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d4eec5f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:58,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:58,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:58,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:58,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47236 deadline: 1681890958993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,000 DEBUG [Thread-1535] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:54:59,000 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,000 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,000 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:59,001 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,001 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:59,002 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,002 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:59,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:59,004 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:59,004 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:59,004 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:59,004 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:59,004 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:59,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:59,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:59,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:59,006 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,008 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,010 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:59,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:59,010 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:59,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:59,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:59,010 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:59,010 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,010 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,010 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,010 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,010 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,010 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:59,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:59,010 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,010 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:59,010 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,110 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:59,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:59,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:59,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:59,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:59,114 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:59,114 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:59,114 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:59,114 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:59,114 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,114 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,116 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:59,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:59,117 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:59,117 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,117 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:59,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:59,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:59,118 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:59,119 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:59,119 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:59,119 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:59,119 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:59,119 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:59,119 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:59,119 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,119 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:59,119 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:59,120 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,120 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:59,121 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,121 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32050ba5[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:59,122 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:59,122 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,124 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:59,124 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,124 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:59,126 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,126 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:59,126 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:59,126 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:59,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:59,128 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:59,128 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:59,128 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:59,128 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:59,128 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:59,130 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,130 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:59,130 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,131 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:59,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,132 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:59,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:59,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:59,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:59,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,134 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:59,134 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:59,135 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:59,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:59,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:59,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:59,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:59,235 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:59,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:59,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:59,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:59,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:59,238 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:59,238 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:59,238 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:59,238 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:59,239 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,239 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,240 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:59,240 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:59,240 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,240 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:59,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:59,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:59,241 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:59,242 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:59,242 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:59,242 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:59,242 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:59,244 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:59,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:59,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,245 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,245 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b611cc6[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:59,246 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:59,246 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,248 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,249 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:59,249 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:59,249 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:59,249 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,249 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:59,250 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:59,250 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,250 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:59,250 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:59,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890959249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:59,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:59,253 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:59,253 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:59,253 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:59,253 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:59,253 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:59,255 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,255 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:59,255 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:59,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:59,255 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,257 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,263 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,263 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,263 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:59,263 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:59,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:59,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:59,263 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:59,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:59,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:59,263 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:59,263 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,263 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,263 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:59,263 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:59,263 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,264 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:59,343 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/52319df6c01f432a89a723757da2a464 2023-04-19 07:54:59,353 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/0fe011845276421a8b077705b1d48fcb 2023-04-19 07:54:59,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/4660cf6cc6924f8fb11ce032ca34676d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/4660cf6cc6924f8fb11ce032ca34676d 2023-04-19 07:54:59,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/4660cf6cc6924f8fb11ce032ca34676d, entries=200, sequenceid=590, filesize=14.3 K 2023-04-19 07:54:59,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/52319df6c01f432a89a723757da2a464 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/52319df6c01f432a89a723757da2a464 2023-04-19 07:54:59,364 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:59,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:59,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:59,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/52319df6c01f432a89a723757da2a464, entries=150, sequenceid=590, filesize=11.9 K 2023-04-19 07:54:59,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:59,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:59,367 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:59,367 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:59,367 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:59,367 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:59,368 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,368 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,369 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:59,369 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:59,370 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,370 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:59,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/0fe011845276421a8b077705b1d48fcb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/0fe011845276421a8b077705b1d48fcb 2023-04-19 07:54:59,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:59,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:59,371 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:59,371 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:59,371 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:59,375 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:59,375 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:59,378 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:59,378 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:59,378 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:59,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/0fe011845276421a8b077705b1d48fcb, entries=150, sequenceid=590, filesize=11.9 K 2023-04-19 07:54:59,380 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:59,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for cfaed3abed1fbb0d5a48467b790cb91c in 859ms, sequenceid=590, compaction requested=true 2023-04-19 07:54:59,381 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:59,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:59,381 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:59,381 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:59,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:59,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:59,382 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51594 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:59,382 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49154 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:59,382 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/A is initiating minor compaction (all files) 2023-04-19 07:54:59,383 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/B is initiating minor compaction (all files) 2023-04-19 07:54:59,383 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/A in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:59,383 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/B in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:59,383 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/088d6a69fb34477aab615c8f16ebeb26, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/df7a62efa0914a7bb9b2cf724209b3b8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/ef1ecbcf894d460c9b049d1ba1e20e84, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/4660cf6cc6924f8fb11ce032ca34676d] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=50.4 K 2023-04-19 07:54:59,383 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/796faeb88eab4e89bc495244a23d35fc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/55df49fc0dd04232aa4dc8497ee9d80d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/50c0bba6e7834f979810d3c7aab148d4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/52319df6c01f432a89a723757da2a464] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=48.0 K 2023-04-19 07:54:59,383 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 088d6a69fb34477aab615c8f16ebeb26, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681890896109 2023-04-19 07:54:59,383 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 796faeb88eab4e89bc495244a23d35fc, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681890896109 2023-04-19 07:54:59,383 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting df7a62efa0914a7bb9b2cf724209b3b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1681890896493 2023-04-19 07:54:59,384 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 55df49fc0dd04232aa4dc8497ee9d80d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1681890896493 2023-04-19 07:54:59,384 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting ef1ecbcf894d460c9b049d1ba1e20e84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681890897346 2023-04-19 07:54:59,384 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 50c0bba6e7834f979810d3c7aab148d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681890897346 2023-04-19 07:54:59,384 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 4660cf6cc6924f8fb11ce032ca34676d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681890897659 2023-04-19 07:54:59,384 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 52319df6c01f432a89a723757da2a464, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681890897659 2023-04-19 07:54:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:54:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:54:59,387 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:54:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:54:59,390 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:59,390 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:59,391 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,391 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:54:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:59,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:59,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:54:59,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a16c2bd[Count = 0] remaining members to acquire global barrier 2023-04-19 07:54:59,393 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:54:59,393 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,393 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#A#compaction#240 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:59,394 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,394 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:54:59,394 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:54:59,394 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,394 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:54:59,394 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:54:59,397 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:54:59,397 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:59,397 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,397 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:54:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:54:59,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:54:59,400 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:54:59,400 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:54:59,400 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:54:59,400 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:54:59,400 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:54:59,403 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,403 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,403 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:59,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:54:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:54:59,403 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:54:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,405 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:54:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:54:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:54:59,408 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:59,408 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:54:59,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:59,408 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,408 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:54:59,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:54:59,408 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,408 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,408 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,408 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:54:59,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:54:59,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:54:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:54:59,409 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:54:59,409 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:54:59,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:54:59,414 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#B#compaction#241 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:59,454 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/5f8e68704cd24e858859b1a55c1e44bf as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/5f8e68704cd24e858859b1a55c1e44bf 2023-04-19 07:54:59,460 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/A of cfaed3abed1fbb0d5a48467b790cb91c into 5f8e68704cd24e858859b1a55c1e44bf(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:59,460 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:59,460 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/A, priority=12, startTime=1681890899381; duration=0sec 2023-04-19 07:54:59,460 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:54:59,460 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:54:59,462 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49154 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:54:59,462 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/C is initiating minor compaction (all files) 2023-04-19 07:54:59,462 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/C in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:59,462 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/ce3fbc0fc9b24a418cecc7d69a6ba164, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/63f2c010f22947ac891f8876aac75630, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/6f12d53494de412c9fcd34096d9213e3, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/0fe011845276421a8b077705b1d48fcb] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=48.0 K 2023-04-19 07:54:59,463 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting ce3fbc0fc9b24a418cecc7d69a6ba164, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681890896109 2023-04-19 07:54:59,463 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 63f2c010f22947ac891f8876aac75630, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1681890896493 2023-04-19 07:54:59,464 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 6f12d53494de412c9fcd34096d9213e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681890897346 2023-04-19 07:54:59,464 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 0fe011845276421a8b077705b1d48fcb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681890897659 2023-04-19 07:54:59,477 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#C#compaction#242 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:54:59,478 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/df921fde5779439e8f785c08a84690ee as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/df921fde5779439e8f785c08a84690ee 2023-04-19 07:54:59,484 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/B of cfaed3abed1fbb0d5a48467b790cb91c into df921fde5779439e8f785c08a84690ee(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:59,485 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:59,485 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/B, priority=12, startTime=1681890899381; duration=0sec 2023-04-19 07:54:59,485 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:59,500 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/54680d081a40453b845c97d0d52384f6 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/54680d081a40453b845c97d0d52384f6 2023-04-19 07:54:59,509 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:54:59,509 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/C of cfaed3abed1fbb0d5a48467b790cb91c into 54680d081a40453b845c97d0d52384f6(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:54:59,510 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:54:59,510 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/C, priority=12, startTime=1681890899381; duration=0sec 2023-04-19 07:54:59,510 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:54:59,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:54:59,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:54:59,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:54:59,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:54:59,513 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:54:59,513 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:59,513 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:54:59,514 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:54:59,514 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,514 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,515 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:54:59,515 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:54:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:54:59,516 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,516 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:54:59,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:54:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:54:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:54:59,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:54:59,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:54:59,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:54:59,517 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:54:59,517 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:54:59,517 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:54:59,520 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:54:59,520 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:54:59,520 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:54:59,520 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:54:59,520 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-19 07:54:59,520 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:54:59,521 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:59,521 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:54:59,521 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:59,521 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:54:59,521 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:54:59,542 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/743c2c9d55914bb196ff3d1b60828168 2023-04-19 07:54:59,575 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/34dbef2ba0074e9780d2357f38fab549 2023-04-19 07:54:59,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:54:59,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:54:59,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:59,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890959775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:59,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890959789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:59,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890959812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:54:59,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:54:59,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890959878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,002 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=630 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/6b3cf74ac54d43ad8c011b20d76dd0ab 2023-04-19 07:55:00,012 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/743c2c9d55914bb196ff3d1b60828168 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/743c2c9d55914bb196ff3d1b60828168 2023-04-19 07:55:00,017 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/743c2c9d55914bb196ff3d1b60828168, entries=150, sequenceid=630, filesize=11.9 K 2023-04-19 07:55:00,018 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/34dbef2ba0074e9780d2357f38fab549 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/34dbef2ba0074e9780d2357f38fab549 2023-04-19 07:55:00,022 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/34dbef2ba0074e9780d2357f38fab549, entries=150, sequenceid=630, filesize=11.9 K 2023-04-19 07:55:00,023 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/6b3cf74ac54d43ad8c011b20d76dd0ab as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/6b3cf74ac54d43ad8c011b20d76dd0ab 2023-04-19 07:55:00,037 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/6b3cf74ac54d43ad8c011b20d76dd0ab, entries=150, sequenceid=630, filesize=11.9 K 2023-04-19 07:55:00,038 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for cfaed3abed1fbb0d5a48467b790cb91c in 518ms, sequenceid=630, compaction requested=false 2023-04-19 07:55:00,038 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:00,038 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,039 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:00,039 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:00,039 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,040 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:00,040 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:00,042 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,042 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,042 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,042 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:00,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7000e3b4[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:00,044 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:00,044 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,045 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,046 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:00,046 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:00,046 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:00,046 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,046 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:00,048 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,048 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,048 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:00,048 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:00,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:00,052 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:00,052 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:00,052 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:00,052 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:00,052 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:00,054 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,054 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,054 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:00,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,055 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:00,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,057 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,060 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:00,060 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:00,060 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:00,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:00,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:00,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:00,060 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:00,060 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,060 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:00,060 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:00,060 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,060 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:00,061 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:00,061 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,061 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:00,082 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 07:55:00,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:55:00,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:00,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:55:00,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:00,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:55:00,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:00,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890900122, max timestamp=1681890900134Num uniques 859; 2023-04-19 07:55:00,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:55:00,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890900122, max timestamp=1681890900134Num uniques 859; 2023-04-19 07:55:00,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890900122, max timestamp=1681890900134Num uniques 859; 2023-04-19 07:55:00,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:55:00,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:55:00,161 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:00,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:00,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:00,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:00,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:00,164 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:00,164 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:00,164 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:00,164 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:00,165 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,165 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,166 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:00,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:00,166 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,167 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,167 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:00,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:00,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:00,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:00,168 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:00,168 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:00,168 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:00,176 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:00,176 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,176 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:00,176 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:00,176 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:00,176 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,176 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:00,176 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:00,176 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,176 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:00,176 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:00,178 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,178 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,178 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,178 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:00,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a428632[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:00,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:00,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,182 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,182 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,182 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,182 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:00,182 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:00,182 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:00,185 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,185 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:00,185 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:00,185 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:00,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:00,188 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:00,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:00,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:00,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:00,188 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:00,189 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,190 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,190 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:00,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:00,190 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:00,192 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890960188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,198 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,198 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,198 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,198 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,198 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,198 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,198 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,198 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:00,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:00,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:00,199 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:00,199 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:00,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:00,199 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:00,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:00,199 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:00,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:00,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:00,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890960293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,299 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:00,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:00,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:00,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:00,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:00,302 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:00,302 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:00,302 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:00,302 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:00,302 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,302 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,304 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,304 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:00,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:00,305 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,305 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:00,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:00,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:00,306 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:00,306 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:00,306 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:00,306 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:00,306 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:00,306 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,306 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:00,306 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:00,306 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,306 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:00,307 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:00,307 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,307 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:00,307 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:00,309 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,309 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,309 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,309 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:00,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49e63090[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:00,311 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:00,311 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,312 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,312 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:00,312 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:00,312 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:00,312 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,312 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:00,313 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,313 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,313 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:00,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:00,316 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:00,316 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:00,316 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:00,316 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:00,316 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:00,322 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,322 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,323 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:00,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,323 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,324 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,326 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,326 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,326 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,326 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,326 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:00,326 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,327 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,327 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:00,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:00,327 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:00,327 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:00,328 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:00,328 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:00,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:00,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:00,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:00,427 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:00,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:00,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:00,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:00,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:00,430 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:00,430 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:00,430 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:00,430 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:00,431 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,431 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,433 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:00,433 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:00,433 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,433 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:00,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:00,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:00,434 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:00,434 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:00,434 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:00,435 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:00,435 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:00,435 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,435 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:00,435 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:00,435 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,435 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:00,435 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:00,435 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,435 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:00,435 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:00,437 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,437 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,437 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,437 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:00,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3599003b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:00,439 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:00,439 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,441 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:00,441 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,441 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:00,443 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,443 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,443 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:00,443 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:00,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:00,446 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:00,446 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:00,446 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:00,446 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:00,446 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:00,448 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,448 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,448 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:00,449 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,450 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,453 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,453 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:00,453 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:00,453 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:00,453 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:00,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:00,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:00,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:00,453 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,453 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,453 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:00,453 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:00,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:00,453 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:00,453 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,453 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,490 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/208cd98c7bc54ef8975282509d104bc5 2023-04-19 07:55:00,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:00,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890960498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/dd743474325942128aa1856d91b68d06 2023-04-19 07:55:00,553 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:00,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:00,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:00,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:00,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:00,557 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:00,557 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:00,557 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:00,557 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:00,558 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,558 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,561 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:00,561 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:00,562 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,562 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:00,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:00,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:00,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:00,563 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:00,563 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,563 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:00,564 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:00,564 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:00,566 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,566 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,566 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,566 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:00,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4273487f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:00,568 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:00,568 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,569 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,569 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:00,569 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:00,569 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:00,569 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,569 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:00,570 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,570 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:00,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:00,573 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:00,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:00,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:00,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:00,573 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:00,576 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,576 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,576 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,576 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,578 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,580 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:00,580 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:00,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:00,580 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:00,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:00,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:00,580 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:00,580 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,581 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,581 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,581 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,581 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,581 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:00,581 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:00,581 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:00,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:00,681 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:00,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:00,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:00,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:00,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:00,684 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:00,684 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:00,684 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:00,684 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:00,685 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,685 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,689 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:00,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:00,689 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,689 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:00,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:00,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:00,691 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,691 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:00,691 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:00,692 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:00,696 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,696 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,696 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,696 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:00,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cfe2515[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:00,698 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:00,698 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,699 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,699 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:00,699 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:00,699 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:00,699 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,699 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:00,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:00,701 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,701 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:00,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:00,703 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:00,703 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:00,703 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:00,703 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:00,703 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:00,706 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,706 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,706 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,707 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,710 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,710 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:00,710 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:00,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:00,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:00,710 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:00,710 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:00,710 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:00,710 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:00,710 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,710 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:00,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:00,710 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,711 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,711 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:00,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890960803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,810 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:00,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:00,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:00,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:00,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:00,815 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:00,815 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:00,815 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:00,815 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:00,815 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,815 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,816 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,816 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:00,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:00,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:00,817 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,817 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:00,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:00,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:00,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:00,818 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,818 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:00,818 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:00,818 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:00,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:00,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:00,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:00,820 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,821 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,821 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:00,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4464e0aa[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:00,822 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:00,822 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,824 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,824 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:00,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:00,824 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:00,824 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:00,826 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:00,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:00,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:00,828 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:00,828 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:00,828 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:00,828 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:00,828 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:00,830 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,830 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:00,830 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:00,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,830 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:00,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:00,831 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,833 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:00,834 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:00,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:00,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:00,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:00,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:00,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:00,834 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:00,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,834 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:00,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:00,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:00,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/4ed88b0441eb4e72935fe9aed15a82e6 2023-04-19 07:55:00,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/208cd98c7bc54ef8975282509d104bc5 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/208cd98c7bc54ef8975282509d104bc5 2023-04-19 07:55:00,918 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/208cd98c7bc54ef8975282509d104bc5, entries=200, sequenceid=641, filesize=14.3 K 2023-04-19 07:55:00,919 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/dd743474325942128aa1856d91b68d06 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/dd743474325942128aa1856d91b68d06 2023-04-19 07:55:00,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/dd743474325942128aa1856d91b68d06, entries=150, sequenceid=641, filesize=11.9 K 2023-04-19 07:55:00,923 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/4ed88b0441eb4e72935fe9aed15a82e6 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/4ed88b0441eb4e72935fe9aed15a82e6 2023-04-19 07:55:00,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/4ed88b0441eb4e72935fe9aed15a82e6, entries=150, sequenceid=641, filesize=11.9 K 2023-04-19 07:55:00,928 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for cfaed3abed1fbb0d5a48467b790cb91c in 846ms, sequenceid=641, compaction requested=true 2023-04-19 07:55:00,928 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:00,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:00,928 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:00,928 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:00,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:00,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:00,929 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:00,929 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:00,929 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/A is initiating minor compaction (all files) 2023-04-19 07:55:00,929 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/B is initiating minor compaction (all files) 2023-04-19 07:55:00,929 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/A in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,929 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/B in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,930 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/5f8e68704cd24e858859b1a55c1e44bf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/743c2c9d55914bb196ff3d1b60828168, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/208cd98c7bc54ef8975282509d104bc5] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=38.6 K 2023-04-19 07:55:00,930 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/df921fde5779439e8f785c08a84690ee, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/34dbef2ba0074e9780d2357f38fab549, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/dd743474325942128aa1856d91b68d06] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=36.2 K 2023-04-19 07:55:00,930 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 5f8e68704cd24e858859b1a55c1e44bf, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681890897659 2023-04-19 07:55:00,930 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting df921fde5779439e8f785c08a84690ee, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681890897659 2023-04-19 07:55:00,930 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 743c2c9d55914bb196ff3d1b60828168, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1681890898628 2023-04-19 07:55:00,931 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 34dbef2ba0074e9780d2357f38fab549, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1681890898628 2023-04-19 07:55:00,931 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 208cd98c7bc54ef8975282509d104bc5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1681890899760 2023-04-19 07:55:00,931 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting dd743474325942128aa1856d91b68d06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1681890899760 2023-04-19 07:55:00,934 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:00,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:00,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:00,938 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#B#compaction#250 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:00,938 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#A#compaction#249 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:00,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:00,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:00,938 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:00,938 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:00,938 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:00,938 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:00,939 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,940 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:00,942 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:00,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,943 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:00,943 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:00,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:00,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:00,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:00,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:00,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:00,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:00,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:00,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:00,952 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:00,952 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:00,952 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:00,952 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:00,952 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-19 07:55:00,952 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:55:00,953 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:00,953 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:55:00,953 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:00,953 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:55:00,953 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:01,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:01,307 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:01,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:01,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890961331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,358 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/ccdaab90dd1a44cf8dbb64c90b551a24 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/ccdaab90dd1a44cf8dbb64c90b551a24 2023-04-19 07:55:01,358 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/d071e98b5a6b480da76ef4dc10a15849 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/d071e98b5a6b480da76ef4dc10a15849 2023-04-19 07:55:01,362 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/6390b169987149c5bfab4498583e375a 2023-04-19 07:55:01,364 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/A of cfaed3abed1fbb0d5a48467b790cb91c into d071e98b5a6b480da76ef4dc10a15849(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:01,364 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:01,364 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/A, priority=13, startTime=1681890900928; duration=0sec 2023-04-19 07:55:01,364 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:01,364 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:01,365 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/B of cfaed3abed1fbb0d5a48467b790cb91c into ccdaab90dd1a44cf8dbb64c90b551a24(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:01,365 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:01,365 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/B, priority=13, startTime=1681890900928; duration=0sec 2023-04-19 07:55:01,365 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:01,367 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:01,367 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/C is initiating minor compaction (all files) 2023-04-19 07:55:01,367 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/C in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:01,367 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/54680d081a40453b845c97d0d52384f6, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/6b3cf74ac54d43ad8c011b20d76dd0ab, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/4ed88b0441eb4e72935fe9aed15a82e6] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=36.2 K 2023-04-19 07:55:01,372 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 54680d081a40453b845c97d0d52384f6, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1681890897659 2023-04-19 07:55:01,373 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 6b3cf74ac54d43ad8c011b20d76dd0ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=630, earliestPutTs=1681890898628 2023-04-19 07:55:01,374 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 4ed88b0441eb4e72935fe9aed15a82e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1681890899760 2023-04-19 07:55:01,386 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#C#compaction#253 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:01,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:01,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890961434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:01,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890961638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,812 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/efda49e35dc845fcb75ab58d455b7f66 2023-04-19 07:55:01,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:01,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890961814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,817 DEBUG [Thread-1537] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:55:01,819 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/56712b36efc4467da652b520d3e19335 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/56712b36efc4467da652b520d3e19335 2023-04-19 07:55:01,823 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/C of cfaed3abed1fbb0d5a48467b790cb91c into 56712b36efc4467da652b520d3e19335(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:01,823 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:01,823 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/C, priority=13, startTime=1681890900928; duration=0sec 2023-04-19 07:55:01,823 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:01,825 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/d241f8529f694e15a778fd9129b073fe 2023-04-19 07:55:01,829 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/6390b169987149c5bfab4498583e375a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/6390b169987149c5bfab4498583e375a 2023-04-19 07:55:01,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:01,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890961828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,831 DEBUG [Thread-1541] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:55:01,833 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/6390b169987149c5bfab4498583e375a, entries=150, sequenceid=677, filesize=11.9 K 2023-04-19 07:55:01,834 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/efda49e35dc845fcb75ab58d455b7f66 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/efda49e35dc845fcb75ab58d455b7f66 2023-04-19 07:55:01,837 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/efda49e35dc845fcb75ab58d455b7f66, entries=150, sequenceid=677, filesize=11.9 K 2023-04-19 07:55:01,838 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/d241f8529f694e15a778fd9129b073fe as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/d241f8529f694e15a778fd9129b073fe 2023-04-19 07:55:01,841 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/d241f8529f694e15a778fd9129b073fe, entries=150, sequenceid=677, filesize=11.9 K 2023-04-19 07:55:01,842 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for cfaed3abed1fbb0d5a48467b790cb91c in 890ms, sequenceid=677, compaction requested=false 2023-04-19 07:55:01,842 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:01,842 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:01,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:01,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:01,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:01,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:01,843 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,843 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:01,844 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,844 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:01,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:01,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:01,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:01,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3981a8c5[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:01,845 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:01,845 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,846 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,847 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:01,847 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:01,847 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,847 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:01,847 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:01,848 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:01,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:01,849 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:01,849 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:01,849 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:01,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:01,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:01,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:01,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:01,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:01,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:01,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:01,851 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:01,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:01,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:01,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:01,852 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:01,854 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,854 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:01,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:01,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,854 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:01,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:01,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:01,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:01,855 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:01,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:01,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:01,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,856 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:01,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:01,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:01,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:01,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:01,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:01,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,859 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:01,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:01,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:01,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:01,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:01,860 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,860 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,860 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:01,860 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:01,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:01,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:01,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:01,943 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 07:55:01,943 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:55:01,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:01,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:55:01,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:01,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:55:01,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:01,960 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:01,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:01,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:01,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:01,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:01,963 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:01,963 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:01,963 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:01,963 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:01,964 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,964 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:01,965 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,965 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:01,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:01,965 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,965 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:01,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:01,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:01,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:01,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:01,966 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:01,966 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:01,966 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:01,967 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:01,967 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:01,967 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:01,967 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:01,967 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:01,969 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,969 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:01,969 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,969 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:01,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:01,971 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:01,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61692f3a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:01,971 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,973 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,973 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:01,973 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:01,973 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,974 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:01,973 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:01,975 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:01,975 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,975 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:01,975 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:01,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:01,978 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:01,978 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:01,978 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:01,978 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:01,978 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:01,980 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,980 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:01,980 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:01,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:01,980 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:01,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:01,981 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:01,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:01,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:01,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:01,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:01,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:01,984 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:01,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:01,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:01,984 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:01,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:01,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:01,984 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:01,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:01,985 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:01,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:01,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:01,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890961989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,084 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:02,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:02,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:02,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:02,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:02,088 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:02,088 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:02,088 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:02,088 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:02,089 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,089 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,090 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:02,090 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:02,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:02,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:02,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:02,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:02,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:02,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:02,092 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:02,092 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:02,092 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:02,092 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:02,092 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:02,092 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,092 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:02,092 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:02,092 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,093 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:02,093 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:02,093 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,093 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:02,093 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:02,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:02,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890962092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,095 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,095 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,095 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,095 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:02,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:02,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cf2688f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:02,097 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:02,097 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,098 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,098 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:02,098 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:02,098 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:02,098 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,098 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:02,100 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,100 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:02,100 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:02,100 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:02,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:02,102 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:02,102 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:02,102 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:02,102 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:02,102 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:02,106 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,106 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,107 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:02,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:02,107 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,108 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,111 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,111 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:02,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:02,111 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:02,111 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:02,111 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:02,111 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,111 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,111 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,111 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,111 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:02,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:02,112 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:02,112 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:02,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:02,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:02,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:02,212 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:02,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:02,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:02,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:02,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:02,215 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:02,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:02,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:02,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:02,216 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,216 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,217 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:02,217 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:02,218 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,218 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:02,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:02,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:02,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:02,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:02,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:02,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:02,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:02,219 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,219 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:02,219 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:02,219 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,220 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:02,220 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:02,220 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,220 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:02,220 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:02,221 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,221 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,222 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,222 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:02,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ff0ac0b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:02,223 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:02,223 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,225 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,225 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:02,225 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:02,225 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:02,225 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,225 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:02,227 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,227 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:02,227 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,227 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:02,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:02,229 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:02,229 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:02,229 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:02,229 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:02,229 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:02,231 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,231 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,231 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:02,232 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,233 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,235 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,235 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,235 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,235 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:02,235 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,235 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:02,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:02,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:02,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:02,235 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:02,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:02,236 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:02,235 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,236 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:02,236 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,236 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:02,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:02,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890962295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,336 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:02,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:02,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:02,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:02,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:02,339 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:02,339 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:02,339 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:02,339 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:02,340 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,340 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,342 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,342 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:02,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:02,342 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,342 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:02,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:02,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:02,343 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:02,343 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:02,343 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:02,343 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:02,343 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:02,344 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,344 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:02,344 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:02,344 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:02,346 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,346 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,346 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,346 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:02,347 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:02,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b03f2b0[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:02,347 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,348 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:02,348 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:02,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:02,348 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:02,353 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,353 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:02,353 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:02,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,354 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,355 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=692 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/e79150dee1ff4ca8a6b959c092623719 2023-04-19 07:55:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:02,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:02,356 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:02,356 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:02,356 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:02,356 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:02,356 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:02,358 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,358 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,358 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,359 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,361 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,363 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:02,363 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,363 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:02,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:02,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:02,363 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:02,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:02,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:02,363 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:02,363 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,364 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,364 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,364 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,364 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:02,364 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:02,364 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:02,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:02,368 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=692 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/15127408e281431492b20b3662d678a4 2023-04-19 07:55:02,464 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:02,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:02,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:02,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:02,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:02,468 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:02,468 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:02,468 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:02,468 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:02,468 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,468 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,471 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,471 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:02,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:02,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:02,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,472 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,472 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:02,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:02,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:02,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:02,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:02,473 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:02,473 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:02,473 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:02,473 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:02,473 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:02,473 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,474 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:02,474 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:02,474 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:02,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:02,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:02,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:02,476 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,476 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,476 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,477 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:02,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:02,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53bb0215[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:02,478 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:02,478 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,479 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,479 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:02,480 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,480 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:02,480 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:02,480 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:02,481 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,481 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:02,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:02,484 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:02,484 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:02,484 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:02,484 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:02,484 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:02,491 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,491 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,492 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:02,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,492 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:02,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:02,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,493 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,496 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,496 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,496 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,496 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:02,496 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:02,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:02,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:02,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:02,496 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:02,496 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,496 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:02,496 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,496 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:02,497 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:02,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:02,497 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:02,597 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:02,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:02,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:02,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890962598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:02,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:02,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:02,600 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:02,600 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:02,600 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:02,600 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:02,601 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,601 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,602 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:02,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:02,602 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,602 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:02,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:02,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:02,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:02,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:02,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:02,604 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,604 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:02,604 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:02,604 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:02,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:02,606 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,606 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,606 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,606 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:02,607 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:02,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c32aefd[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:02,607 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,609 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,609 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:02,610 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,610 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:02,611 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,611 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:02,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,611 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:02,611 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:02,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:02,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:02,613 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:02,613 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:02,613 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:02,613 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:02,613 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:02,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:02,616 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,617 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,619 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:02,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:02,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:02,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:02,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:02,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,620 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:02,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:02,620 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,620 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:02,620 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:02,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:02,720 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:02,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:02,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:02,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:02,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:02,724 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:02,724 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:02,724 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:02,724 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:02,724 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,725 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,727 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:02,727 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:02,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:02,727 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,727 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:02,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:02,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:02,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:02,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:02,729 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,729 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:02,729 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:02,729 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:02,731 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,731 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,731 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,731 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:02,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:02,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26e8c784[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:02,733 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:02,733 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,734 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,734 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,734 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:02,734 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:02,735 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:02,735 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:02,736 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,736 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:02,736 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,736 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:02,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:02,738 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:02,738 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:02,738 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:02,739 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:02,739 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:02,741 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,741 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,742 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:02,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:02,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:02,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,742 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:02,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:02,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:02,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,743 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,748 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:02,748 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:02,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:02,748 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:02,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:02,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:02,748 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:02,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:02,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:02,748 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,749 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:02,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:02,749 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:02,749 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,749 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,749 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,749 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:02,749 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,778 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=692 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/db71cd384da34449b0aa70c0c612a0fb 2023-04-19 07:55:02,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/e79150dee1ff4ca8a6b959c092623719 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e79150dee1ff4ca8a6b959c092623719 2023-04-19 07:55:02,785 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e79150dee1ff4ca8a6b959c092623719, entries=200, sequenceid=692, filesize=14.3 K 2023-04-19 07:55:02,786 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/15127408e281431492b20b3662d678a4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/15127408e281431492b20b3662d678a4 2023-04-19 07:55:02,789 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/15127408e281431492b20b3662d678a4, entries=150, sequenceid=692, filesize=11.9 K 2023-04-19 07:55:02,790 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/db71cd384da34449b0aa70c0c612a0fb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/db71cd384da34449b0aa70c0c612a0fb 2023-04-19 07:55:02,793 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/db71cd384da34449b0aa70c0c612a0fb, entries=150, sequenceid=692, filesize=11.9 K 2023-04-19 07:55:02,794 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for cfaed3abed1fbb0d5a48467b790cb91c in 851ms, sequenceid=692, compaction requested=true 2023-04-19 07:55:02,794 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:02,794 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:02,794 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:02,794 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:02,794 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:02,794 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:02,795 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:02,795 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:02,795 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/A is initiating minor compaction (all files) 2023-04-19 07:55:02,795 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/B is initiating minor compaction (all files) 2023-04-19 07:55:02,795 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/A in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,795 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/B in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,795 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/d071e98b5a6b480da76ef4dc10a15849, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/6390b169987149c5bfab4498583e375a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e79150dee1ff4ca8a6b959c092623719] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=38.7 K 2023-04-19 07:55:02,795 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/ccdaab90dd1a44cf8dbb64c90b551a24, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/efda49e35dc845fcb75ab58d455b7f66, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/15127408e281431492b20b3662d678a4] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=36.3 K 2023-04-19 07:55:02,796 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting d071e98b5a6b480da76ef4dc10a15849, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1681890899760 2023-04-19 07:55:02,796 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting ccdaab90dd1a44cf8dbb64c90b551a24, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1681890899760 2023-04-19 07:55:02,796 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 6390b169987149c5bfab4498583e375a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1681890900167 2023-04-19 07:55:02,796 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting efda49e35dc845fcb75ab58d455b7f66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1681890900167 2023-04-19 07:55:02,796 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting e79150dee1ff4ca8a6b959c092623719, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1681890901319 2023-04-19 07:55:02,797 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 15127408e281431492b20b3662d678a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1681890901326 2023-04-19 07:55:02,802 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#B#compaction#259 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:02,803 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#A#compaction#258 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:02,849 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:02,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:02,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:02,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:02,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:02,852 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:02,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:02,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:02,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:02,857 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,857 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,858 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:02,859 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:02,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:02,859 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:02,859 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:02,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:02,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:02,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:02,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:02,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:02,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:02,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:02,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:02,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:02,861 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:02,861 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:02,861 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 07:55:02,861 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:55:02,861 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:02,861 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:55:02,861 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:02,861 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:55:02,861 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:03,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:03,105 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:03,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:03,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890963130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:03,212 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/e9643f53c0d44c078cc6f2b2245f1c0f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/e9643f53c0d44c078cc6f2b2245f1c0f 2023-04-19 07:55:03,220 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/a7724936d31a41a6b38d5fa10c1b6793 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/a7724936d31a41a6b38d5fa10c1b6793 2023-04-19 07:55:03,224 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/B of cfaed3abed1fbb0d5a48467b790cb91c into e9643f53c0d44c078cc6f2b2245f1c0f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:03,224 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:03,224 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/B, priority=13, startTime=1681890902794; duration=0sec 2023-04-19 07:55:03,224 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:03,224 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:03,225 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:03,225 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/C is initiating minor compaction (all files) 2023-04-19 07:55:03,225 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/C in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:03,225 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/56712b36efc4467da652b520d3e19335, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/d241f8529f694e15a778fd9129b073fe, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/db71cd384da34449b0aa70c0c612a0fb] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=36.3 K 2023-04-19 07:55:03,226 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 56712b36efc4467da652b520d3e19335, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1681890899760 2023-04-19 07:55:03,226 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting d241f8529f694e15a778fd9129b073fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1681890900167 2023-04-19 07:55:03,226 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/A of cfaed3abed1fbb0d5a48467b790cb91c into a7724936d31a41a6b38d5fa10c1b6793(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:03,226 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting db71cd384da34449b0aa70c0c612a0fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1681890901326 2023-04-19 07:55:03,226 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:03,226 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/A, priority=13, startTime=1681890902794; duration=0sec 2023-04-19 07:55:03,227 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:03,231 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#C#compaction#261 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:03,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:03,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890963234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:03,269 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/69f1344630a6444d8b7821931ed61347 2023-04-19 07:55:03,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:03,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890963438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:03,642 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/58243ed8782e477b8590a6575c71eebc as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/58243ed8782e477b8590a6575c71eebc 2023-04-19 07:55:03,648 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/C of cfaed3abed1fbb0d5a48467b790cb91c into 58243ed8782e477b8590a6575c71eebc(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:03,648 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:03,648 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/C, priority=13, startTime=1681890902794; duration=0sec 2023-04-19 07:55:03,648 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:03,683 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/a39745f4b1ed4359aa8dd8b97f37983f 2023-04-19 07:55:03,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:03,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890963741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,093 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/efec1f880c874c52b59c4a61f4e811bb 2023-04-19 07:55:04,097 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/69f1344630a6444d8b7821931ed61347 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/69f1344630a6444d8b7821931ed61347 2023-04-19 07:55:04,101 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/69f1344630a6444d8b7821931ed61347, entries=150, sequenceid=712, filesize=11.9 K 2023-04-19 07:55:04,101 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/a39745f4b1ed4359aa8dd8b97f37983f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/a39745f4b1ed4359aa8dd8b97f37983f 2023-04-19 07:55:04,105 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/a39745f4b1ed4359aa8dd8b97f37983f, entries=150, sequenceid=712, filesize=11.9 K 2023-04-19 07:55:04,106 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/efec1f880c874c52b59c4a61f4e811bb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/efec1f880c874c52b59c4a61f4e811bb 2023-04-19 07:55:04,109 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/efec1f880c874c52b59c4a61f4e811bb, entries=150, sequenceid=712, filesize=11.9 K 2023-04-19 07:55:04,110 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for cfaed3abed1fbb0d5a48467b790cb91c in 1249ms, sequenceid=712, compaction requested=false 2023-04-19 07:55:04,110 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:04,110 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:04,110 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:04,110 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:04,110 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,110 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:04,110 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:04,113 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,113 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:04,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:04,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:04,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:04,113 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:04,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:04,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:04,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:04,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:04,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:04,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@429a279b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:04,115 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:04,115 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:04,116 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:04,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:04,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:04,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:04,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:04,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:04,116 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,116 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:04,118 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:04,118 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:04,118 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,118 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:04,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:04,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:04,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:04,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:04,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:04,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:04,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:04,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:04,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:04,120 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:04,120 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:04,120 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:04,121 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:04,121 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:04,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:04,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:04,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:04,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:04,123 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:04,124 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:04,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:04,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:04,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:04,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,128 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:04,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:04,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:04,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:04,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:04,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:04,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:04,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:04,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:04,128 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:04,130 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:04,130 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:04,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:04,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:04,230 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:04,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:04,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:04,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:04,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:04,234 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:04,234 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:04,234 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:04,234 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:04,235 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:04,235 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:04,236 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:04,236 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:04,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:04,237 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,237 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:04,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:04,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:04,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:04,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:04,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:04,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:04,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:04,238 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:04,238 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:04,238 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:04,238 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:04,238 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:04,238 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:04,238 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:04,238 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 07:55:04,238 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:55:04,238 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:04,238 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:55:04,238 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:04,239 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:55:04,239 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:04,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:04,249 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:04,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:04,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890964296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:04,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890964400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:04,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890964603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:04,648 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/7cb5ce1f56164958a589864961edef1b 2023-04-19 07:55:04,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:04,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890964905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,059 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/8f70680bee7d41b29cd03c145e5fe745 2023-04-19 07:55:05,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:05,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890965408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,472 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/1623a47b8cb943de8679ca737597fc02 2023-04-19 07:55:05,480 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/7cb5ce1f56164958a589864961edef1b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/7cb5ce1f56164958a589864961edef1b 2023-04-19 07:55:05,484 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/7cb5ce1f56164958a589864961edef1b, entries=150, sequenceid=727, filesize=11.9 K 2023-04-19 07:55:05,484 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/8f70680bee7d41b29cd03c145e5fe745 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/8f70680bee7d41b29cd03c145e5fe745 2023-04-19 07:55:05,488 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/8f70680bee7d41b29cd03c145e5fe745, entries=150, sequenceid=727, filesize=11.9 K 2023-04-19 07:55:05,489 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/1623a47b8cb943de8679ca737597fc02 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/1623a47b8cb943de8679ca737597fc02 2023-04-19 07:55:05,492 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/1623a47b8cb943de8679ca737597fc02, entries=150, sequenceid=727, filesize=11.9 K 2023-04-19 07:55:05,493 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for cfaed3abed1fbb0d5a48467b790cb91c in 1255ms, sequenceid=727, compaction requested=true 2023-04-19 07:55:05,493 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:05,493 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:05,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:05,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:05,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:05,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:05,495 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:05,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:05,495 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:05,495 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:05,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:05,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49ba3e77[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:05,497 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:05,497 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:05,498 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:05,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:05,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:05,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:05,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:05,498 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:05,498 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,498 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:05,500 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,500 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:05,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:05,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:05,500 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:05,501 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:05,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:05,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:05,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:05,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:05,503 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:05,503 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:05,503 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:05,503 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:05,503 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:05,504 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:05,504 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:05,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:05,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:05,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:05,504 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:05,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:05,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:05,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:05,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:05,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:05,505 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:05,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,509 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,509 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:05,509 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:05,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:05,509 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:05,509 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:05,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:05,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:05,509 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,509 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:05,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:05,509 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:05,509 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:05,509 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:05,509 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:05,509 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:05,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:05,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:05,609 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:05,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:05,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:05,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:05,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:05,613 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:05,613 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:05,613 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:05,613 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:05,613 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:05,613 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:05,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:05,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:05,615 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:05,615 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,615 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:05,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:05,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:05,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:05,616 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:05,617 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:05,617 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:05,617 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:05,617 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:05,617 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:05,617 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:05,617 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 07:55:05,617 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:55:05,618 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:05,618 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:55:05,618 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:05,618 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:55:05,618 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:05,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:05,849 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:05,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:05,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890965867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:05,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890965868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:05,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890965970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:05,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:05,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890965971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,030 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=747 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/920f01fbca2d4f639a3a12673c824163 2023-04-19 07:55:06,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:06,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890966172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:06,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890966175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:06,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890966417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,441 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=747 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/286df829a47441dbaa6cc9564f8f3cf4 2023-04-19 07:55:06,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:06,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890966474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:06,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890966477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,854 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=747 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/3346d2661310481599e57418e034bdc8 2023-04-19 07:55:06,858 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/920f01fbca2d4f639a3a12673c824163 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/920f01fbca2d4f639a3a12673c824163 2023-04-19 07:55:06,863 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/920f01fbca2d4f639a3a12673c824163, entries=150, sequenceid=747, filesize=11.9 K 2023-04-19 07:55:06,864 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/286df829a47441dbaa6cc9564f8f3cf4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/286df829a47441dbaa6cc9564f8f3cf4 2023-04-19 07:55:06,879 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/286df829a47441dbaa6cc9564f8f3cf4, entries=150, sequenceid=747, filesize=11.9 K 2023-04-19 07:55:06,879 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/3346d2661310481599e57418e034bdc8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/3346d2661310481599e57418e034bdc8 2023-04-19 07:55:06,883 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/3346d2661310481599e57418e034bdc8, entries=150, sequenceid=747, filesize=11.9 K 2023-04-19 07:55:06,884 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for cfaed3abed1fbb0d5a48467b790cb91c in 1267ms, sequenceid=747, compaction requested=true 2023-04-19 07:55:06,884 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:06,884 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:06,884 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:06,884 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:06,885 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,885 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:06,885 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:06,887 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:06,887 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:06,887 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:06,887 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:06,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:06,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@300634fb[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:06,889 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:06,889 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:06,891 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:06,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:06,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:06,891 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,891 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:06,891 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:06,891 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:06,891 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:06,893 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,893 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:06,893 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:06,893 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:06,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:06,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:06,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:06,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:06,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:06,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:06,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:06,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:06,896 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:06,896 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:06,896 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:06,896 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:06,896 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:06,899 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:06,899 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:06,899 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:06,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:06,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:06,900 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,901 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:06,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:06,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:06,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:06,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:06,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:06,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:06,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:06,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:06,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:06,908 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:06,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:06,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:06,908 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:06,909 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:06,909 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:06,909 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:06,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:06,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:06,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:06,983 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-19 07:55:06,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:55:06,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:06,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:55:06,983 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:06,983 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:55:06,984 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:06,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=760 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/92023241ecb548ec8676d6c83f07fcf8 2023-04-19 07:55:07,008 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=760 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/5c831b1bcce9431cabfdfa458d64ad06 2023-04-19 07:55:07,009 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:07,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:07,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:07,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:07,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:07,014 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:07,014 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:07,014 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:07,014 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:07,014 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,015 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,016 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:07,016 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:07,016 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,016 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:07,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:07,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:07,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:07,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:07,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:07,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:07,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:07,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890967025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:07,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890967026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,032 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:07,032 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:07,032 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:07,032 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:07,032 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:07,032 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:07,032 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:07,032 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:07,032 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,032 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:07,032 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:07,034 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,034 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:07,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:07,034 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,034 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:07,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:07,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:07,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,036 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=760 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/55dbcb675b0c4c8281fe7c6b0b668b7f 2023-04-19 07:55:07,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:07,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:07,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50b21198[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:07,037 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:07,037 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,039 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,039 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:07,039 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:07,039 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:07,041 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:07,041 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,042 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:07,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/92023241ecb548ec8676d6c83f07fcf8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/92023241ecb548ec8676d6c83f07fcf8 2023-04-19 07:55:07,042 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:07,042 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:07,042 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:07,042 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:07,042 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:07,042 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:07,044 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,044 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:07,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,044 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:07,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:07,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:07,045 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,046 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/92023241ecb548ec8676d6c83f07fcf8, entries=200, sequenceid=760, filesize=14.3 K 2023-04-19 07:55:07,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:07,046 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/5c831b1bcce9431cabfdfa458d64ad06 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/5c831b1bcce9431cabfdfa458d64ad06 2023-04-19 07:55:07,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:07,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:07,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:07,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:07,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:07,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:07,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:07,050 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:07,050 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:07,051 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/5c831b1bcce9431cabfdfa458d64ad06, entries=150, sequenceid=760, filesize=11.9 K 2023-04-19 07:55:07,051 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:07,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:07,051 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/55dbcb675b0c4c8281fe7c6b0b668b7f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/55dbcb675b0c4c8281fe7c6b0b668b7f 2023-04-19 07:55:07,055 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/55dbcb675b0c4c8281fe7c6b0b668b7f, entries=150, sequenceid=760, filesize=11.9 K 2023-04-19 07:55:07,055 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for cfaed3abed1fbb0d5a48467b790cb91c in 72ms, sequenceid=760, compaction requested=true 2023-04-19 07:55:07,055 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:07,055 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:07,056 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:55:07,056 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:55:07,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:07,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:07,057 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64120 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:55:07,057 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/A is initiating minor compaction (all files) 2023-04-19 07:55:07,057 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/A in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:07,057 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/a7724936d31a41a6b38d5fa10c1b6793, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/69f1344630a6444d8b7821931ed61347, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/7cb5ce1f56164958a589864961edef1b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/920f01fbca2d4f639a3a12673c824163, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/92023241ecb548ec8676d6c83f07fcf8] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=62.6 K 2023-04-19 07:55:07,058 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:55:07,058 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting a7724936d31a41a6b38d5fa10c1b6793, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1681890901326 2023-04-19 07:55:07,058 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/B is initiating minor compaction (all files) 2023-04-19 07:55:07,058 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/B in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:07,058 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/e9643f53c0d44c078cc6f2b2245f1c0f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/a39745f4b1ed4359aa8dd8b97f37983f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/8f70680bee7d41b29cd03c145e5fe745, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/286df829a47441dbaa6cc9564f8f3cf4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/5c831b1bcce9431cabfdfa458d64ad06] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=60.2 K 2023-04-19 07:55:07,058 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 69f1344630a6444d8b7821931ed61347, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=712, earliestPutTs=1681890901979 2023-04-19 07:55:07,058 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting e9643f53c0d44c078cc6f2b2245f1c0f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1681890901326 2023-04-19 07:55:07,058 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 7cb5ce1f56164958a589864961edef1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1681890903117 2023-04-19 07:55:07,059 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting a39745f4b1ed4359aa8dd8b97f37983f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=712, earliestPutTs=1681890901979 2023-04-19 07:55:07,059 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 920f01fbca2d4f639a3a12673c824163, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=747, earliestPutTs=1681890904285 2023-04-19 07:55:07,059 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 92023241ecb548ec8676d6c83f07fcf8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1681890905861 2023-04-19 07:55:07,059 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 8f70680bee7d41b29cd03c145e5fe745, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1681890903117 2023-04-19 07:55:07,060 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 286df829a47441dbaa6cc9564f8f3cf4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=747, earliestPutTs=1681890904285 2023-04-19 07:55:07,061 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 5c831b1bcce9431cabfdfa458d64ad06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1681890905861 2023-04-19 07:55:07,071 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#A#compaction#274 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:07,071 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#B#compaction#273 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:07,083 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/aa26085ee6d84818ba6e8c21d265d1dc as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/aa26085ee6d84818ba6e8c21d265d1dc 2023-04-19 07:55:07,088 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/B of cfaed3abed1fbb0d5a48467b790cb91c into aa26085ee6d84818ba6e8c21d265d1dc(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:07,088 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:07,088 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/B, priority=11, startTime=1681890907056; duration=0sec 2023-04-19 07:55:07,088 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:07,088 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:55:07,089 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:55:07,090 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/C is initiating minor compaction (all files) 2023-04-19 07:55:07,090 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/C in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:07,090 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/58243ed8782e477b8590a6575c71eebc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/efec1f880c874c52b59c4a61f4e811bb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/1623a47b8cb943de8679ca737597fc02, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/3346d2661310481599e57418e034bdc8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/55dbcb675b0c4c8281fe7c6b0b668b7f] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=60.2 K 2023-04-19 07:55:07,090 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 58243ed8782e477b8590a6575c71eebc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1681890901326 2023-04-19 07:55:07,090 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting efec1f880c874c52b59c4a61f4e811bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=712, earliestPutTs=1681890901979 2023-04-19 07:55:07,091 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 1623a47b8cb943de8679ca737597fc02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1681890903117 2023-04-19 07:55:07,091 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 3346d2661310481599e57418e034bdc8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=747, earliestPutTs=1681890904285 2023-04-19 07:55:07,091 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 55dbcb675b0c4c8281fe7c6b0b668b7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1681890905861 2023-04-19 07:55:07,099 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#C#compaction#275 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:07,105 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/4320729861f14e79acf45a734bec4edc as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/4320729861f14e79acf45a734bec4edc 2023-04-19 07:55:07,109 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/C of cfaed3abed1fbb0d5a48467b790cb91c into 4320729861f14e79acf45a734bec4edc(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:07,110 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:07,110 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/C, priority=11, startTime=1681890907056; duration=0sec 2023-04-19 07:55:07,110 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:07,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:07,131 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 07:55:07,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:55:07,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:07,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:55:07,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:07,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:55:07,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:07,144 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=782 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/83fee41ffce14543904162bcb818ea41 2023-04-19 07:55:07,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:07,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890967145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:07,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890967148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,151 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:07,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:07,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:07,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:07,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:07,160 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:07,160 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:07,160 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:07,160 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:07,161 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,161 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:07,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:07,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:07,163 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,164 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,164 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:07,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:07,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:07,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:07,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:07,168 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:07,168 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:07,168 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:07,168 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:07,168 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:07,168 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:07,168 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:07,168 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:07,169 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:07,169 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:07,169 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:07,169 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,169 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:07,169 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:07,170 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,170 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:07,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:07,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:07,171 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,171 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:07,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:07,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:07,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:07,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@430752ea[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:07,173 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:07,173 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,174 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,174 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,174 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:07,174 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:07,174 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:07,174 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:07,176 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:07,176 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:07,176 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,176 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:07,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:07,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:07,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:07,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:07,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:07,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:07,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:07,179 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:07,179 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:07,179 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:07,179 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:07,179 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:07,181 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,181 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,182 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:07,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:07,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:07,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:07,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:07,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,183 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,184 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:07,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,186 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:07,186 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:07,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:07,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:07,186 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:07,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:07,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:07,186 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,187 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:07,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:07,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:07,187 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:07,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:07,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890967249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:07,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890967251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,287 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:07,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:07,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:07,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:07,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:07,291 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:07,291 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:07,291 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:07,291 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:07,291 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,291 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,294 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:07,294 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:07,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:07,294 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,294 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:07,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:07,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:07,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:07,296 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:07,296 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:07,296 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:07,296 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:07,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:07,298 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,298 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:07,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:07,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:07,298 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,298 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:07,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:07,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:07,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:07,300 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:07,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bd6857b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:07,300 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,302 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:07,302 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:07,302 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:07,302 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,302 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:07,304 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,304 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:07,304 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:07,304 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:07,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:07,307 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:07,307 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:07,307 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:07,307 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:07,307 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:07,309 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,309 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,309 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:07,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:07,309 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,311 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:07,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,314 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,314 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,314 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:07,314 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:07,314 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:07,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:07,314 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:07,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:07,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:07,314 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,314 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,314 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:07,314 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:07,314 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,314 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:07,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:07,414 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:07,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:07,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:07,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:07,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:07,418 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:07,418 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:07,418 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:07,418 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:07,419 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,419 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,423 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,423 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:07,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:07,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:07,423 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,423 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:07,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:07,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:07,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:07,425 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:07,425 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:07,425 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:07,425 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:07,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:07,427 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,427 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:07,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:07,427 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:07,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:07,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:07,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:07,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:07,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5af6c316[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:07,429 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:07,429 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,430 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,430 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:07,430 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:07,430 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:07,430 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,431 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:07,432 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:07,433 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:07,433 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:07,433 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:07,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:07,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:07,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:07,435 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:07,435 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:07,435 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:07,435 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:07,435 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:07,437 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,437 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:07,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:07,437 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:07,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,438 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:07,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:07,439 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,441 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,441 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,441 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:07,441 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:07,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:07,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:07,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:07,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:07,441 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:07,441 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:07,442 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:07,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:07,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:07,442 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,442 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:07,442 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,442 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,442 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:07,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890967453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:07,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890967455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,484 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/1fb5ee980aca4f4f9b1e59c224c64c62 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/1fb5ee980aca4f4f9b1e59c224c64c62 2023-04-19 07:55:07,489 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/A of cfaed3abed1fbb0d5a48467b790cb91c into 1fb5ee980aca4f4f9b1e59c224c64c62(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:07,489 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:07,489 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/A, priority=11, startTime=1681890907055; duration=0sec 2023-04-19 07:55:07,489 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:07,542 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:07,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:07,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:07,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:07,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:07,546 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:07,546 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:07,546 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:07,546 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:07,546 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,546 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,548 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:07,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:07,548 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,548 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,548 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:07,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:07,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:07,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:07,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:07,549 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:07,549 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:07,549 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:07,549 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:07,549 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:07,549 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:07,549 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:07,550 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:07,550 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:07,550 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:07,550 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:07,550 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,550 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:07,550 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:07,552 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,552 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:07,553 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,553 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:07,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:07,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:07,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a883701[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:07,554 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:07,555 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,556 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,556 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:07,556 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:07,556 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:07,556 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,556 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:07,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:07,558 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=782 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/8e2cdd22736e45bfaa181ebe2296f1d4 2023-04-19 07:55:07,558 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:07,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:07,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:07,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:07,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:07,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:07,562 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:07,562 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:07,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:07,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:07,563 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:07,565 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,565 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,565 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:07,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:07,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,566 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:07,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:07,568 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:07,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:07,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:07,571 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=782 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/76ffed16f6f345ddafc2f66f2a4fa956 2023-04-19 07:55:07,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:07,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:07,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:07,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:07,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:07,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:07,572 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:07,573 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:07,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,573 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,573 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,573 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,573 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:07,573 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,573 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:07,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:07,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:07,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:07,576 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/83fee41ffce14543904162bcb818ea41 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/83fee41ffce14543904162bcb818ea41 2023-04-19 07:55:07,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/83fee41ffce14543904162bcb818ea41, entries=150, sequenceid=782, filesize=11.9 K 2023-04-19 07:55:07,582 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/8e2cdd22736e45bfaa181ebe2296f1d4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/8e2cdd22736e45bfaa181ebe2296f1d4 2023-04-19 07:55:07,587 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/8e2cdd22736e45bfaa181ebe2296f1d4, entries=150, sequenceid=782, filesize=11.9 K 2023-04-19 07:55:07,588 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/76ffed16f6f345ddafc2f66f2a4fa956 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/76ffed16f6f345ddafc2f66f2a4fa956 2023-04-19 07:55:07,592 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/76ffed16f6f345ddafc2f66f2a4fa956, entries=150, sequenceid=782, filesize=11.9 K 2023-04-19 07:55:07,593 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for cfaed3abed1fbb0d5a48467b790cb91c in 462ms, sequenceid=782, compaction requested=false 2023-04-19 07:55:07,593 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:07,673 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:07,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:07,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:07,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:07,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:07,677 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:07,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:07,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:07,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:07,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,678 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:07,679 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:07,679 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,679 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:07,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:07,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:07,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:07,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:07,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:07,681 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 07:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:55:07,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:07,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:07,757 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:07,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:07,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890967779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:07,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890967781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:07,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890967883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:07,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:07,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890967884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:08,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890968085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,090 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/e9f5e782785e47baa393c8f2c96c45d4 2023-04-19 07:55:08,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:08,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890968088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:08,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890968389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:08,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890968394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:08,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47244 deadline: 1681890968427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,430 DEBUG [Thread-1539] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4133 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:55:08,503 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/1606ba62706947debe95429420af2b7d 2023-04-19 07:55:08,514 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/cf33e80b900540a9a78b98ab3574d366 2023-04-19 07:55:08,518 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/e9f5e782785e47baa393c8f2c96c45d4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e9f5e782785e47baa393c8f2c96c45d4 2023-04-19 07:55:08,521 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e9f5e782785e47baa393c8f2c96c45d4, entries=150, sequenceid=795, filesize=11.9 K 2023-04-19 07:55:08,522 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/1606ba62706947debe95429420af2b7d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/1606ba62706947debe95429420af2b7d 2023-04-19 07:55:08,525 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/1606ba62706947debe95429420af2b7d, entries=150, sequenceid=795, filesize=11.9 K 2023-04-19 07:55:08,526 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/cf33e80b900540a9a78b98ab3574d366 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cf33e80b900540a9a78b98ab3574d366 2023-04-19 07:55:08,529 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cf33e80b900540a9a78b98ab3574d366, entries=150, sequenceid=795, filesize=11.9 K 2023-04-19 07:55:08,530 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for cfaed3abed1fbb0d5a48467b790cb91c in 848ms, sequenceid=795, compaction requested=true 2023-04-19 07:55:08,530 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:08,530 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:08,530 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:08,530 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:08,530 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,530 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:08,530 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:08,531 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,531 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:08,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:08,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:08,532 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:08,532 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:08,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:08,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:08,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:08,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:08,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:08,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63e1b6b8[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:08,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:08,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:08,534 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:08,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:08,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:08,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:08,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:08,534 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,535 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:08,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:08,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:08,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:08,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:08,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:08,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:08,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:08,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:08,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:08,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:08,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:08,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:08,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:08,538 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:08,538 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:08,538 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:08,538 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:08,538 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:08,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:08,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:08,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:08,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:08,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:08,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:08,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:08,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:08,542 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:08,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:08,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:08,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:08,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:08,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:08,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:08,544 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:08,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:08,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:08,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:08,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:08,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:08,546 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:08,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:08,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:08,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:08,546 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:08,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:08,546 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:08,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:08,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:08,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:08,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:08,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:08,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:08,646 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:08,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:08,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:08,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:08,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:08,650 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:08,650 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:08,650 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:08,650 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:08,650 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:08,650 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:08,651 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,651 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:08,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:08,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:08,652 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,652 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:08,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:08,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:08,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:08,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:08,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:08,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:08,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:08,653 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:08,653 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:08,653 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:08,653 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:08,653 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:08,653 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:08,653 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:08,653 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 07:55:08,653 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:55:08,654 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:08,654 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:55:08,654 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:08,654 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:55:08,654 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:08,660 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=815 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/55f6b030a358401194f220fb1fad981c 2023-04-19 07:55:08,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:08,895 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:08,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:08,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890968909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:08,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:08,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890968910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:09,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890969012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:09,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890969013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:09,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47286 deadline: 1681890969051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,053 DEBUG [Thread-1533] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18230 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:55:09,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:09,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47236 deadline: 1681890969057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,061 DEBUG [Thread-1535] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18239 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:55:09,071 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=815 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/708a7084caca41818280c33a8212cf42 2023-04-19 07:55:09,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:09,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890969214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:09,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890969217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,483 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=815 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/56b67d57e4c34a08ab12711af1cc8fc7 2023-04-19 07:55:09,487 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/55f6b030a358401194f220fb1fad981c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/55f6b030a358401194f220fb1fad981c 2023-04-19 07:55:09,491 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/55f6b030a358401194f220fb1fad981c, entries=150, sequenceid=815, filesize=11.9 K 2023-04-19 07:55:09,491 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/708a7084caca41818280c33a8212cf42 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/708a7084caca41818280c33a8212cf42 2023-04-19 07:55:09,495 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/708a7084caca41818280c33a8212cf42, entries=150, sequenceid=815, filesize=11.9 K 2023-04-19 07:55:09,495 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/56b67d57e4c34a08ab12711af1cc8fc7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/56b67d57e4c34a08ab12711af1cc8fc7 2023-04-19 07:55:09,498 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/56b67d57e4c34a08ab12711af1cc8fc7, entries=150, sequenceid=815, filesize=11.9 K 2023-04-19 07:55:09,499 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for cfaed3abed1fbb0d5a48467b790cb91c in 846ms, sequenceid=815, compaction requested=true 2023-04-19 07:55:09,499 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:09,499 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:09,499 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:09,499 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:09,499 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,499 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:09,499 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:09,504 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,504 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:09,505 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,505 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:09,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:09,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cf31ae2[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:09,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:09,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,508 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,508 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:09,508 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:09,508 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,508 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:09,508 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:09,510 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:09,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:09,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:09,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:09,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:09,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:09,512 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:09,512 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:09,512 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:09,512 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:09,512 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:09,514 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,514 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:09,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,514 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:09,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:09,515 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,516 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,519 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:09,519 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:09,519 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:09,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:09,519 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,519 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:09,520 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,520 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:09,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:09,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:09,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:09,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:09,520 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:09,520 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-19 07:55:09,520 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,520 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,520 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:55:09,520 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:09,521 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:55:09,521 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:09,521 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:55:09,521 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:09,522 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:09,523 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:09,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:09,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890969547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:09,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890969548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,623 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:09,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:09,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:09,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:09,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:09,626 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:09,626 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:09,626 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:09,626 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:09,627 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,627 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,628 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:09,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:09,628 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,628 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,628 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:09,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:09,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:09,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:09,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:09,630 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:09,630 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:09,630 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:09,630 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:09,631 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,631 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:09,632 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:09,632 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:09,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:09,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60719619[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:09,633 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:09,633 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,634 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,634 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:09,634 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:09,634 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:09,635 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,635 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:09,637 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:09,637 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,637 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:09,637 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:09,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:09,639 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:09,639 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:09,639 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:09,639 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:09,639 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:09,640 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,640 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:09,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,640 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:09,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:09,641 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:09,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,642 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:09,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,644 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,644 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,644 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,644 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:09,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:09,644 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:09,644 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:09,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:09,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:09,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:09,644 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:09,645 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,645 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,645 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,645 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:09,645 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:09,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:09,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:09,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:09,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890969651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:09,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890969652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,745 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:09,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:09,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:09,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:09,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:09,748 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:09,748 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:09,748 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:09,748 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:09,750 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,750 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,751 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:09,751 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:09,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:09,752 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,752 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:09,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:09,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:09,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:09,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:09,753 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:09,753 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:09,753 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:09,753 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:09,753 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:09,753 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:09,753 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:09,753 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:09,753 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:09,753 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:09,753 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:09,753 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,753 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:09,753 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:09,756 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:09,756 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:09,756 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,756 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:09,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:09,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b9a8529[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:09,757 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:09,758 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,759 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,759 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:09,759 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:09,759 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:09,759 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,759 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:09,760 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,760 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:09,760 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:09,760 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:09,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:09,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:09,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:09,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:09,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:09,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:09,763 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:09,763 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:09,763 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:09,763 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:09,763 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:09,764 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,764 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,764 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:09,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,765 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:09,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:09,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:09,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:09,766 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:09,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,769 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,769 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,769 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:09,769 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:09,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:09,769 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:09,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:09,769 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:09,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:09,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:09,769 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:09,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:09,770 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,770 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,770 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,770 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:09,771 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:09,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:09,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890969855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:09,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890969856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,871 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:09,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:09,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:09,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:09,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:09,874 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:09,874 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:09,874 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:09,874 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:09,875 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,875 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,876 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:09,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:09,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:09,876 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,877 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,877 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:09,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:09,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:09,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:09,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:09,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:09,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:09,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:09,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:09,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:09,878 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:09,878 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:09,878 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:09,878 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:09,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:09,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:09,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:09,878 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:09,880 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:09,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:09,880 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:09,880 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,881 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:09,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:09,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:09,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:09,882 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:09,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55a0a094[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:09,882 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,883 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,883 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:09,883 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:09,883 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:09,883 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,883 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:09,888 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,888 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:09,888 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:09,888 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:09,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:09,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:09,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:09,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:09,890 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:09,890 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:09,890 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:09,890 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:09,890 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:09,892 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,892 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:09,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,892 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:09,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:09,892 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:09,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:09,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:09,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:09,894 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:09,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:09,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:09,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,896 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:09,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:09,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:09,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:09,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:09,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:09,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:09,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:09,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:09,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:09,896 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:09,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,896 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:09,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:09,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:09,896 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:09,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=828 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/915ee91ac4514de8a5323a49e7f7279a 2023-04-19 07:55:09,996 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:09,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:09,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:10,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:10,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:10,000 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:10,000 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:10,000 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:10,000 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:10,000 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,001 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,002 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:10,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:10,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:10,002 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,002 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,003 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:10,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:10,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:10,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:10,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:10,003 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:10,003 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:10,003 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:10,003 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:10,004 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:10,004 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:10,004 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:10,004 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:10,004 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:10,004 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:10,004 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:10,004 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,004 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:10,004 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:10,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,006 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:10,006 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,007 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:10,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:10,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:10,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:10,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:10,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@419084c8[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:10,008 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:10,008 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,009 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,009 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:10,009 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:10,009 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:10,010 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,010 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:10,011 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:10,011 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:10,011 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:10,011 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:10,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:10,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:10,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:10,013 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:10,014 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:10,014 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:10,014 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:10,014 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:10,015 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,015 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:10,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,016 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:10,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:10,016 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:10,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,017 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,022 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,022 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,022 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,022 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,022 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,022 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,022 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:10,023 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:10,023 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:10,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:10,023 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:10,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:10,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:10,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:10,023 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:10,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:10,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:10,023 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:10,123 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:10,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:10,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:10,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:10,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:10,128 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:10,128 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:10,128 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:10,128 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:10,128 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,128 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,129 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,129 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:10,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:10,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:10,130 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,130 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:10,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:10,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:10,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:10,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:10,131 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:10,131 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:10,131 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:10,131 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:10,132 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:10,132 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:10,132 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:10,132 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:10,132 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:10,132 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:10,132 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:10,132 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,132 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:10,132 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:10,133 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,133 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:10,134 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,134 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:10,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:10,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:10,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:10,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:10,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d3d13bd[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:10,135 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:10,135 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,136 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,137 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:10,137 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:10,137 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:10,137 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,137 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:10,139 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:10,139 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:10,139 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,139 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:10,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:10,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:10,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:10,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:10,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:10,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:10,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:10,142 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:10,142 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:10,142 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:10,142 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:10,142 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:10,144 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,144 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:10,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,144 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:10,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:10,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:10,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:10,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:10,145 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:10,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,146 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:10,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,148 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,148 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,148 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,148 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:10,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:10,148 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:10,148 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:10,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:10,148 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:10,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:10,148 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:10,149 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,149 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:10,149 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,149 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:10,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:10,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:10,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:10,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890970158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:10,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890970159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,249 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:10,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:10,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:10,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:10,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:10,252 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:10,252 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:10,252 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:10,252 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:10,253 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,253 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,254 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,254 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:10,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:10,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:10,254 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,254 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:10,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:10,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:10,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:10,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:10,255 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:10,255 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:10,255 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:10,255 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:10,256 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:10,256 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:10,256 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. started... 2023-04-19 07:55:10,256 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. as already flushing 2023-04-19 07:55:10,256 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:10,256 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:10,256 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:10,256 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,256 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:10,256 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:10,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,258 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:10,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:10,259 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:10,259 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:10,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:10,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:10,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:10,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@426168fe[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:10,260 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:10,260 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,261 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,262 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:10,262 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:10,262 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:10,262 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,262 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:10,263 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:10,263 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,263 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:10,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:10,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:10,263 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:10,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:10,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:10,265 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:10,265 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:10,265 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:10,266 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:10,266 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:10,267 DEBUG [Thread-1548] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x087b3095 to 127.0.0.1:58087 2023-04-19 07:55:10,267 DEBUG [Thread-1548] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:10,268 DEBUG [Thread-1546] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ce5573e to 127.0.0.1:58087 2023-04-19 07:55:10,268 DEBUG [Thread-1546] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:10,269 DEBUG [Thread-1552] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e65578e to 127.0.0.1:58087 2023-04-19 07:55:10,269 DEBUG [Thread-1552] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:10,270 DEBUG [Thread-1550] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12115052 to 127.0.0.1:58087 2023-04-19 07:55:10,270 DEBUG [Thread-1550] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:10,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,271 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:10,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,271 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:10,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:10,271 DEBUG [Thread-1544] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6919b2a0 to 127.0.0.1:58087 2023-04-19 07:55:10,271 DEBUG [Thread-1544] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:10,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:10,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:10,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,273 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:10,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:10,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,275 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,276 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:10,276 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:10,276 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:10,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:10,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:10,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:10,276 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,276 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:10,276 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,276 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,276 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:10,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:10,276 DEBUG [Thread-1543] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:10,276 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:10,276 DEBUG [Thread-1543] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:10,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=828 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/b87017014cab4d629f5464dbca0cdceb 2023-04-19 07:55:10,376 DEBUG [Thread-1543] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:10,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:10,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:10,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47214 deadline: 1681890970661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:10,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47274 deadline: 1681890970662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:10,751 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=828 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/311c0d201de14d1fbf4ca08eda03b7de 2023-04-19 07:55:10,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/915ee91ac4514de8a5323a49e7f7279a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/915ee91ac4514de8a5323a49e7f7279a 2023-04-19 07:55:10,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/915ee91ac4514de8a5323a49e7f7279a, entries=200, sequenceid=828, filesize=14.3 K 2023-04-19 07:55:10,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/b87017014cab4d629f5464dbca0cdceb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/b87017014cab4d629f5464dbca0cdceb 2023-04-19 07:55:10,761 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/b87017014cab4d629f5464dbca0cdceb, entries=150, sequenceid=828, filesize=11.9 K 2023-04-19 07:55:10,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/311c0d201de14d1fbf4ca08eda03b7de as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/311c0d201de14d1fbf4ca08eda03b7de 2023-04-19 07:55:10,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/311c0d201de14d1fbf4ca08eda03b7de, entries=150, sequenceid=828, filesize=11.9 K 2023-04-19 07:55:10,765 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for cfaed3abed1fbb0d5a48467b790cb91c in 1245ms, sequenceid=828, compaction requested=true 2023-04-19 07:55:10,765 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:10,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:10,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:10,765 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:55:10,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:10,765 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:55:10,767 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64290 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:55:10,767 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:55:10,767 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/A is initiating minor compaction (all files) 2023-04-19 07:55:10,767 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/B is initiating minor compaction (all files) 2023-04-19 07:55:10,767 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/A in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:10,767 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/B in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:10,767 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/1fb5ee980aca4f4f9b1e59c224c64c62, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/83fee41ffce14543904162bcb818ea41, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e9f5e782785e47baa393c8f2c96c45d4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/55f6b030a358401194f220fb1fad981c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/915ee91ac4514de8a5323a49e7f7279a] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=62.8 K 2023-04-19 07:55:10,767 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/aa26085ee6d84818ba6e8c21d265d1dc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/8e2cdd22736e45bfaa181ebe2296f1d4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/1606ba62706947debe95429420af2b7d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/708a7084caca41818280c33a8212cf42, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/b87017014cab4d629f5464dbca0cdceb] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=60.4 K 2023-04-19 07:55:10,767 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 1fb5ee980aca4f4f9b1e59c224c64c62, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1681890905861 2023-04-19 07:55:10,767 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting aa26085ee6d84818ba6e8c21d265d1dc, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1681890905861 2023-04-19 07:55:10,767 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 83fee41ffce14543904162bcb818ea41, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=782, earliestPutTs=1681890906994 2023-04-19 07:55:10,768 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 8e2cdd22736e45bfaa181ebe2296f1d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=782, earliestPutTs=1681890906994 2023-04-19 07:55:10,768 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting e9f5e782785e47baa393c8f2c96c45d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681890907137 2023-04-19 07:55:10,768 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 1606ba62706947debe95429420af2b7d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681890907137 2023-04-19 07:55:10,768 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 55f6b030a358401194f220fb1fad981c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1681890907777 2023-04-19 07:55:10,768 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 708a7084caca41818280c33a8212cf42, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1681890907777 2023-04-19 07:55:10,768 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 915ee91ac4514de8a5323a49e7f7279a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=828, earliestPutTs=1681890908899 2023-04-19 07:55:10,769 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting b87017014cab4d629f5464dbca0cdceb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=828, earliestPutTs=1681890908899 2023-04-19 07:55:10,777 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#A#compaction#288 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:10,779 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#B#compaction#289 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:11,186 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/a3941249907d4b2d9c76cb9ceb9630ff as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/a3941249907d4b2d9c76cb9ceb9630ff 2023-04-19 07:55:11,187 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/b226736eafd548e4ac68b05e3a485f1d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/b226736eafd548e4ac68b05e3a485f1d 2023-04-19 07:55:11,189 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/A of cfaed3abed1fbb0d5a48467b790cb91c into a3941249907d4b2d9c76cb9ceb9630ff(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:11,189 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:11,189 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/A, priority=11, startTime=1681890910765; duration=0sec 2023-04-19 07:55:11,189 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:11,189 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:55:11,190 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/B of cfaed3abed1fbb0d5a48467b790cb91c into b226736eafd548e4ac68b05e3a485f1d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:11,190 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:11,190 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/B, priority=11, startTime=1681890910765; duration=0sec 2023-04-19 07:55:11,190 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:11,190 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:55:11,191 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): cfaed3abed1fbb0d5a48467b790cb91c/C is initiating minor compaction (all files) 2023-04-19 07:55:11,191 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cfaed3abed1fbb0d5a48467b790cb91c/C in TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:11,191 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/4320729861f14e79acf45a734bec4edc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/76ffed16f6f345ddafc2f66f2a4fa956, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cf33e80b900540a9a78b98ab3574d366, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/56b67d57e4c34a08ab12711af1cc8fc7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/311c0d201de14d1fbf4ca08eda03b7de] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp, totalSize=60.4 K 2023-04-19 07:55:11,191 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 4320729861f14e79acf45a734bec4edc, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=760, earliestPutTs=1681890905861 2023-04-19 07:55:11,191 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 76ffed16f6f345ddafc2f66f2a4fa956, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=782, earliestPutTs=1681890906994 2023-04-19 07:55:11,191 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting cf33e80b900540a9a78b98ab3574d366, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681890907137 2023-04-19 07:55:11,192 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 56b67d57e4c34a08ab12711af1cc8fc7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1681890907777 2023-04-19 07:55:11,192 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 311c0d201de14d1fbf4ca08eda03b7de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=828, earliestPutTs=1681890908899 2023-04-19 07:55:11,199 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): cfaed3abed1fbb0d5a48467b790cb91c#C#compaction#290 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:11,606 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/8ce9f973850a4e61b2e3ed4297d20abb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/8ce9f973850a4e61b2e3ed4297d20abb 2023-04-19 07:55:11,610 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cfaed3abed1fbb0d5a48467b790cb91c/C of cfaed3abed1fbb0d5a48467b790cb91c into 8ce9f973850a4e61b2e3ed4297d20abb(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:11,610 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:11,610 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c., storeName=cfaed3abed1fbb0d5a48467b790cb91c/C, priority=11, startTime=1681890910765; duration=0sec 2023-04-19 07:55:11,610 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:11,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:11,667 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 07:55:11,667 DEBUG [Thread-1537] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00ca898c to 127.0.0.1:58087 2023-04-19 07:55:11,667 DEBUG [Thread-1537] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:11,667 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:55:11,667 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:11,667 DEBUG [Thread-1541] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7480f9da to 127.0.0.1:58087 2023-04-19 07:55:11,667 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:55:11,668 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:11,668 DEBUG [Thread-1541] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:11,668 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:55:11,668 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:12,074 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=852 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/3367d1bdac7f45518cdfe3902d1a7ef6 2023-04-19 07:55:12,470 DEBUG [Thread-1539] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6585a70a to 127.0.0.1:58087 2023-04-19 07:55:12,470 DEBUG [Thread-1539] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:12,484 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=852 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/2f30121cc2954da99c4d4825772d4859 2023-04-19 07:55:12,892 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=852 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/cd7a3e201be84f1fbdb2264d601d0b5a 2023-04-19 07:55:12,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/3367d1bdac7f45518cdfe3902d1a7ef6 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/3367d1bdac7f45518cdfe3902d1a7ef6 2023-04-19 07:55:12,898 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/3367d1bdac7f45518cdfe3902d1a7ef6, entries=150, sequenceid=852, filesize=11.9 K 2023-04-19 07:55:12,899 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/2f30121cc2954da99c4d4825772d4859 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/2f30121cc2954da99c4d4825772d4859 2023-04-19 07:55:12,902 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/2f30121cc2954da99c4d4825772d4859, entries=150, sequenceid=852, filesize=11.9 K 2023-04-19 07:55:12,903 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/cd7a3e201be84f1fbdb2264d601d0b5a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cd7a3e201be84f1fbdb2264d601d0b5a 2023-04-19 07:55:12,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cd7a3e201be84f1fbdb2264d601d0b5a, entries=150, sequenceid=852, filesize=11.9 K 2023-04-19 07:55:12,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=6.71 KB/6870 for cfaed3abed1fbb0d5a48467b790cb91c in 1239ms, sequenceid=852, compaction requested=false 2023-04-19 07:55:12,906 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:19,104 DEBUG [Thread-1533] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x22b4eae5 to 127.0.0.1:58087 2023-04-19 07:55:19,104 DEBUG [Thread-1533] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:19,142 DEBUG [Thread-1535] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x262f4570 to 127.0.0.1:58087 2023-04-19 07:55:19,142 DEBUG [Thread-1535] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:19,142 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-19 07:55:19,142 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 49 2023-04-19 07:55:19,142 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 69 2023-04-19 07:55:19,142 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 210 2023-04-19 07:55:19,142 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 283 2023-04-19 07:55:19,142 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 141 2023-04-19 07:55:19,142 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-19 07:55:19,142 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(398): read 5416 2023-04-19 07:55:19,142 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(398): read 5213 2023-04-19 07:55:19,142 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(398): read 5226 2023-04-19 07:55:19,143 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(398): read 5430 2023-04-19 07:55:19,143 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(398): read 5211 2023-04-19 07:55:19,143 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-19 07:55:19,143 INFO [Listener at localhost/42517] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-19 07:55:19,143 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x458eac52 to 127.0.0.1:58087 2023-04-19 07:55:19,143 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:19,143 INFO [Listener at localhost/42517] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-19 07:55:19,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-19 07:55:19,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:19,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-19 07:55:19,147 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890919147"}]},"ts":"1681890919147"} 2023-04-19 07:55:19,148 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-19 07:55:19,151 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-19 07:55:19,151 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cfaed3abed1fbb0d5a48467b790cb91c, UNASSIGN}] 2023-04-19 07:55:19,153 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cfaed3abed1fbb0d5a48467b790cb91c, UNASSIGN 2023-04-19 07:55:19,153 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=cfaed3abed1fbb0d5a48467b790cb91c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:19,153 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890919153"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890919153"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890919153"}]},"ts":"1681890919153"} 2023-04-19 07:55:19,154 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:55:19,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-19 07:55:19,307 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:19,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing cfaed3abed1fbb0d5a48467b790cb91c, disabling compactions & flushes 2023-04-19 07:55:19,307 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:19,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:19,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. after waiting 0 ms 2023-04-19 07:55:19,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:19,307 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing cfaed3abed1fbb0d5a48467b790cb91c 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-19 07:55:19,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=A 2023-04-19 07:55:19,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:19,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=B 2023-04-19 07:55:19,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:19,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cfaed3abed1fbb0d5a48467b790cb91c, store=C 2023-04-19 07:55:19,308 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:19,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-19 07:55:19,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=858 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/a853d8bfb8094629a56221ad80046b04 2023-04-19 07:55:19,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-19 07:55:20,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=858 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/70981e901cc344f7b73898e262f5567c 2023-04-19 07:55:20,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-19 07:55:20,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=858 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/bdb694f67056454ba12b1e37da4e4a46 2023-04-19 07:55:20,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/A/a853d8bfb8094629a56221ad80046b04 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/a853d8bfb8094629a56221ad80046b04 2023-04-19 07:55:20,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/a853d8bfb8094629a56221ad80046b04, entries=150, sequenceid=858, filesize=11.9 K 2023-04-19 07:55:20,543 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/B/70981e901cc344f7b73898e262f5567c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/70981e901cc344f7b73898e262f5567c 2023-04-19 07:55:20,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/70981e901cc344f7b73898e262f5567c, entries=150, sequenceid=858, filesize=11.9 K 2023-04-19 07:55:20,547 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/.tmp/C/bdb694f67056454ba12b1e37da4e4a46 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/bdb694f67056454ba12b1e37da4e4a46 2023-04-19 07:55:20,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/bdb694f67056454ba12b1e37da4e4a46, entries=150, sequenceid=858, filesize=11.9 K 2023-04-19 07:55:20,552 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for cfaed3abed1fbb0d5a48467b790cb91c in 1245ms, sequenceid=858, compaction requested=true 2023-04-19 07:55:20,585 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/990964db056a47ebbfdf93750f2bda4d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/6b8487b5f6614f8fb38de11a9d6891a9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/1d5ad41481224c9a84c2b0e91b0058bf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/2976b925835d494dacfbd460bfa387f4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/12f00c3c56c94c1e984cb229f8e34f6e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/4bb93a4c6998434b821785ae9b8fe3e2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e24d5bf9de9640caaa8c2fabe1053aa1, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/13ff2cc14f434265afdb8036286726bc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/c23ad1d32f3247a894b49cd5336a94c0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/eadc4b6a18a24821b00e405d13c25bf5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/974cf33b9a454130909ebe1991a97a96, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/74b5f1bda1244e7ba48ca2486be938e6, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/088d6a69fb34477aab615c8f16ebeb26, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/df7a62efa0914a7bb9b2cf724209b3b8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/ef1ecbcf894d460c9b049d1ba1e20e84, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/4660cf6cc6924f8fb11ce032ca34676d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/5f8e68704cd24e858859b1a55c1e44bf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/743c2c9d55914bb196ff3d1b60828168, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/208cd98c7bc54ef8975282509d104bc5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/d071e98b5a6b480da76ef4dc10a15849, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/6390b169987149c5bfab4498583e375a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e79150dee1ff4ca8a6b959c092623719, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/a7724936d31a41a6b38d5fa10c1b6793, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/69f1344630a6444d8b7821931ed61347, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/7cb5ce1f56164958a589864961edef1b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/920f01fbca2d4f639a3a12673c824163, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/92023241ecb548ec8676d6c83f07fcf8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/1fb5ee980aca4f4f9b1e59c224c64c62, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/83fee41ffce14543904162bcb818ea41, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e9f5e782785e47baa393c8f2c96c45d4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/55f6b030a358401194f220fb1fad981c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/915ee91ac4514de8a5323a49e7f7279a] to archive 2023-04-19 07:55:20,587 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:55:20,590 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/990964db056a47ebbfdf93750f2bda4d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/990964db056a47ebbfdf93750f2bda4d 2023-04-19 07:55:20,592 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/6b8487b5f6614f8fb38de11a9d6891a9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/6b8487b5f6614f8fb38de11a9d6891a9 2023-04-19 07:55:20,593 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/1d5ad41481224c9a84c2b0e91b0058bf to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/1d5ad41481224c9a84c2b0e91b0058bf 2023-04-19 07:55:20,594 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/2976b925835d494dacfbd460bfa387f4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/2976b925835d494dacfbd460bfa387f4 2023-04-19 07:55:20,597 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/12f00c3c56c94c1e984cb229f8e34f6e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/12f00c3c56c94c1e984cb229f8e34f6e 2023-04-19 07:55:20,598 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/4bb93a4c6998434b821785ae9b8fe3e2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/4bb93a4c6998434b821785ae9b8fe3e2 2023-04-19 07:55:20,600 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e24d5bf9de9640caaa8c2fabe1053aa1 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e24d5bf9de9640caaa8c2fabe1053aa1 2023-04-19 07:55:20,602 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/13ff2cc14f434265afdb8036286726bc to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/13ff2cc14f434265afdb8036286726bc 2023-04-19 07:55:20,603 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/c23ad1d32f3247a894b49cd5336a94c0 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/c23ad1d32f3247a894b49cd5336a94c0 2023-04-19 07:55:20,604 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/eadc4b6a18a24821b00e405d13c25bf5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/eadc4b6a18a24821b00e405d13c25bf5 2023-04-19 07:55:20,608 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/974cf33b9a454130909ebe1991a97a96 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/974cf33b9a454130909ebe1991a97a96 2023-04-19 07:55:20,609 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/74b5f1bda1244e7ba48ca2486be938e6 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/74b5f1bda1244e7ba48ca2486be938e6 2023-04-19 07:55:20,610 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/088d6a69fb34477aab615c8f16ebeb26 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/088d6a69fb34477aab615c8f16ebeb26 2023-04-19 07:55:20,612 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/df7a62efa0914a7bb9b2cf724209b3b8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/df7a62efa0914a7bb9b2cf724209b3b8 2023-04-19 07:55:20,614 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/ef1ecbcf894d460c9b049d1ba1e20e84 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/ef1ecbcf894d460c9b049d1ba1e20e84 2023-04-19 07:55:20,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/4660cf6cc6924f8fb11ce032ca34676d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/4660cf6cc6924f8fb11ce032ca34676d 2023-04-19 07:55:20,616 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/5f8e68704cd24e858859b1a55c1e44bf to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/5f8e68704cd24e858859b1a55c1e44bf 2023-04-19 07:55:20,617 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/743c2c9d55914bb196ff3d1b60828168 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/743c2c9d55914bb196ff3d1b60828168 2023-04-19 07:55:20,618 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/208cd98c7bc54ef8975282509d104bc5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/208cd98c7bc54ef8975282509d104bc5 2023-04-19 07:55:20,620 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/d071e98b5a6b480da76ef4dc10a15849 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/d071e98b5a6b480da76ef4dc10a15849 2023-04-19 07:55:20,621 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/6390b169987149c5bfab4498583e375a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/6390b169987149c5bfab4498583e375a 2023-04-19 07:55:20,621 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e79150dee1ff4ca8a6b959c092623719 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e79150dee1ff4ca8a6b959c092623719 2023-04-19 07:55:20,622 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/a7724936d31a41a6b38d5fa10c1b6793 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/a7724936d31a41a6b38d5fa10c1b6793 2023-04-19 07:55:20,624 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/69f1344630a6444d8b7821931ed61347 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/69f1344630a6444d8b7821931ed61347 2023-04-19 07:55:20,625 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/7cb5ce1f56164958a589864961edef1b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/7cb5ce1f56164958a589864961edef1b 2023-04-19 07:55:20,626 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/920f01fbca2d4f639a3a12673c824163 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/920f01fbca2d4f639a3a12673c824163 2023-04-19 07:55:20,626 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/92023241ecb548ec8676d6c83f07fcf8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/92023241ecb548ec8676d6c83f07fcf8 2023-04-19 07:55:20,627 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/1fb5ee980aca4f4f9b1e59c224c64c62 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/1fb5ee980aca4f4f9b1e59c224c64c62 2023-04-19 07:55:20,628 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/83fee41ffce14543904162bcb818ea41 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/83fee41ffce14543904162bcb818ea41 2023-04-19 07:55:20,629 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e9f5e782785e47baa393c8f2c96c45d4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/e9f5e782785e47baa393c8f2c96c45d4 2023-04-19 07:55:20,630 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/55f6b030a358401194f220fb1fad981c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/55f6b030a358401194f220fb1fad981c 2023-04-19 07:55:20,631 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/915ee91ac4514de8a5323a49e7f7279a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/915ee91ac4514de8a5323a49e7f7279a 2023-04-19 07:55:20,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7128a63e249b4b4192e8b679e4c9a146, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/66841a259cd842db8ef92f44cf587df7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/9edbb119ada84340ba2e4f7824b91381, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/f20233317cda4a4c880978d72831f676, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/d5911e040aa742b49793db1dbca9ef29, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7b88c50217604ea99e5cf6fb8ff515b4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/e73f89c1828d4cc6b46a1f98346f0bc2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/cb5f0bfd751c413486a64fe441cb33db, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/33032b91ab9546718acc1fc760c57984, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/5c81d91cc8fb436ea50aa7d3ff737219, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7078f71258fd437aa8f1fdfd61f18b50, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/796faeb88eab4e89bc495244a23d35fc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/53916dfb4b454864bf96cfdce5e1d68c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/55df49fc0dd04232aa4dc8497ee9d80d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/50c0bba6e7834f979810d3c7aab148d4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/df921fde5779439e8f785c08a84690ee, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/52319df6c01f432a89a723757da2a464, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/34dbef2ba0074e9780d2357f38fab549, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/ccdaab90dd1a44cf8dbb64c90b551a24, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/dd743474325942128aa1856d91b68d06, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/efda49e35dc845fcb75ab58d455b7f66, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/e9643f53c0d44c078cc6f2b2245f1c0f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/15127408e281431492b20b3662d678a4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/a39745f4b1ed4359aa8dd8b97f37983f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/8f70680bee7d41b29cd03c145e5fe745, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/286df829a47441dbaa6cc9564f8f3cf4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/aa26085ee6d84818ba6e8c21d265d1dc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/5c831b1bcce9431cabfdfa458d64ad06, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/8e2cdd22736e45bfaa181ebe2296f1d4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/1606ba62706947debe95429420af2b7d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/708a7084caca41818280c33a8212cf42, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/b87017014cab4d629f5464dbca0cdceb] to archive 2023-04-19 07:55:20,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:55:20,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7128a63e249b4b4192e8b679e4c9a146 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7128a63e249b4b4192e8b679e4c9a146 2023-04-19 07:55:20,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/66841a259cd842db8ef92f44cf587df7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/66841a259cd842db8ef92f44cf587df7 2023-04-19 07:55:20,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/9edbb119ada84340ba2e4f7824b91381 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/9edbb119ada84340ba2e4f7824b91381 2023-04-19 07:55:20,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/f20233317cda4a4c880978d72831f676 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/f20233317cda4a4c880978d72831f676 2023-04-19 07:55:20,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/d5911e040aa742b49793db1dbca9ef29 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/d5911e040aa742b49793db1dbca9ef29 2023-04-19 07:55:20,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7b88c50217604ea99e5cf6fb8ff515b4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7b88c50217604ea99e5cf6fb8ff515b4 2023-04-19 07:55:20,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/e73f89c1828d4cc6b46a1f98346f0bc2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/e73f89c1828d4cc6b46a1f98346f0bc2 2023-04-19 07:55:20,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/cb5f0bfd751c413486a64fe441cb33db to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/cb5f0bfd751c413486a64fe441cb33db 2023-04-19 07:55:20,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/33032b91ab9546718acc1fc760c57984 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/33032b91ab9546718acc1fc760c57984 2023-04-19 07:55:20,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/5c81d91cc8fb436ea50aa7d3ff737219 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/5c81d91cc8fb436ea50aa7d3ff737219 2023-04-19 07:55:20,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7078f71258fd437aa8f1fdfd61f18b50 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/7078f71258fd437aa8f1fdfd61f18b50 2023-04-19 07:55:20,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/796faeb88eab4e89bc495244a23d35fc to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/796faeb88eab4e89bc495244a23d35fc 2023-04-19 07:55:20,648 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/53916dfb4b454864bf96cfdce5e1d68c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/53916dfb4b454864bf96cfdce5e1d68c 2023-04-19 07:55:20,649 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/55df49fc0dd04232aa4dc8497ee9d80d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/55df49fc0dd04232aa4dc8497ee9d80d 2023-04-19 07:55:20,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/50c0bba6e7834f979810d3c7aab148d4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/50c0bba6e7834f979810d3c7aab148d4 2023-04-19 07:55:20,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/df921fde5779439e8f785c08a84690ee to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/df921fde5779439e8f785c08a84690ee 2023-04-19 07:55:20,652 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/52319df6c01f432a89a723757da2a464 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/52319df6c01f432a89a723757da2a464 2023-04-19 07:55:20,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/34dbef2ba0074e9780d2357f38fab549 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/34dbef2ba0074e9780d2357f38fab549 2023-04-19 07:55:20,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/ccdaab90dd1a44cf8dbb64c90b551a24 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/ccdaab90dd1a44cf8dbb64c90b551a24 2023-04-19 07:55:20,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/dd743474325942128aa1856d91b68d06 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/dd743474325942128aa1856d91b68d06 2023-04-19 07:55:20,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/efda49e35dc845fcb75ab58d455b7f66 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/efda49e35dc845fcb75ab58d455b7f66 2023-04-19 07:55:20,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/e9643f53c0d44c078cc6f2b2245f1c0f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/e9643f53c0d44c078cc6f2b2245f1c0f 2023-04-19 07:55:20,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/15127408e281431492b20b3662d678a4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/15127408e281431492b20b3662d678a4 2023-04-19 07:55:20,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/a39745f4b1ed4359aa8dd8b97f37983f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/a39745f4b1ed4359aa8dd8b97f37983f 2023-04-19 07:55:20,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/8f70680bee7d41b29cd03c145e5fe745 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/8f70680bee7d41b29cd03c145e5fe745 2023-04-19 07:55:20,660 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/286df829a47441dbaa6cc9564f8f3cf4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/286df829a47441dbaa6cc9564f8f3cf4 2023-04-19 07:55:20,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/aa26085ee6d84818ba6e8c21d265d1dc to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/aa26085ee6d84818ba6e8c21d265d1dc 2023-04-19 07:55:20,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/5c831b1bcce9431cabfdfa458d64ad06 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/5c831b1bcce9431cabfdfa458d64ad06 2023-04-19 07:55:20,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/8e2cdd22736e45bfaa181ebe2296f1d4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/8e2cdd22736e45bfaa181ebe2296f1d4 2023-04-19 07:55:20,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/1606ba62706947debe95429420af2b7d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/1606ba62706947debe95429420af2b7d 2023-04-19 07:55:20,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/708a7084caca41818280c33a8212cf42 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/708a7084caca41818280c33a8212cf42 2023-04-19 07:55:20,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/b87017014cab4d629f5464dbca0cdceb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/b87017014cab4d629f5464dbca0cdceb 2023-04-19 07:55:20,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/51bf9d43fbe84353b20a06297fd7105d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/dff4b51069a4486ab11f8d3aa6cd9411, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cf79064f7b364df8838a7c129ff79e96, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/c42cac8819944f3b925e047944c9fc34, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/7e22d0ab9d9d4916928b9c427b277453, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/69576c2ad1df4f36beb19e30348da8b8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/bbab1ffb7e0446cbb067c8cc54d5cf0f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/14d4899ba4204510b09f1193b4a3ace9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/31a3b439d6624363bef9f771bada13bc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/5a69ac8b56464f62bccca8fc1fc4cc8d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/147aec96eea244509d724a735b1cd74b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/ce3fbc0fc9b24a418cecc7d69a6ba164, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/dbabd0a9ab4c4c9f97340f34806e01d4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/63f2c010f22947ac891f8876aac75630, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/6f12d53494de412c9fcd34096d9213e3, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/54680d081a40453b845c97d0d52384f6, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/0fe011845276421a8b077705b1d48fcb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/6b3cf74ac54d43ad8c011b20d76dd0ab, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/56712b36efc4467da652b520d3e19335, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/4ed88b0441eb4e72935fe9aed15a82e6, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/d241f8529f694e15a778fd9129b073fe, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/58243ed8782e477b8590a6575c71eebc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/db71cd384da34449b0aa70c0c612a0fb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/efec1f880c874c52b59c4a61f4e811bb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/1623a47b8cb943de8679ca737597fc02, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/3346d2661310481599e57418e034bdc8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/4320729861f14e79acf45a734bec4edc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/55dbcb675b0c4c8281fe7c6b0b668b7f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/76ffed16f6f345ddafc2f66f2a4fa956, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cf33e80b900540a9a78b98ab3574d366, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/56b67d57e4c34a08ab12711af1cc8fc7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/311c0d201de14d1fbf4ca08eda03b7de] to archive 2023-04-19 07:55:20,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:55:20,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/51bf9d43fbe84353b20a06297fd7105d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/51bf9d43fbe84353b20a06297fd7105d 2023-04-19 07:55:20,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/dff4b51069a4486ab11f8d3aa6cd9411 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/dff4b51069a4486ab11f8d3aa6cd9411 2023-04-19 07:55:20,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cf79064f7b364df8838a7c129ff79e96 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cf79064f7b364df8838a7c129ff79e96 2023-04-19 07:55:20,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/c42cac8819944f3b925e047944c9fc34 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/c42cac8819944f3b925e047944c9fc34 2023-04-19 07:55:20,674 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/7e22d0ab9d9d4916928b9c427b277453 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/7e22d0ab9d9d4916928b9c427b277453 2023-04-19 07:55:20,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/69576c2ad1df4f36beb19e30348da8b8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/69576c2ad1df4f36beb19e30348da8b8 2023-04-19 07:55:20,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/bbab1ffb7e0446cbb067c8cc54d5cf0f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/bbab1ffb7e0446cbb067c8cc54d5cf0f 2023-04-19 07:55:20,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/14d4899ba4204510b09f1193b4a3ace9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/14d4899ba4204510b09f1193b4a3ace9 2023-04-19 07:55:20,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/31a3b439d6624363bef9f771bada13bc to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/31a3b439d6624363bef9f771bada13bc 2023-04-19 07:55:20,678 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/5a69ac8b56464f62bccca8fc1fc4cc8d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/5a69ac8b56464f62bccca8fc1fc4cc8d 2023-04-19 07:55:20,679 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/147aec96eea244509d724a735b1cd74b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/147aec96eea244509d724a735b1cd74b 2023-04-19 07:55:20,680 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/ce3fbc0fc9b24a418cecc7d69a6ba164 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/ce3fbc0fc9b24a418cecc7d69a6ba164 2023-04-19 07:55:20,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/dbabd0a9ab4c4c9f97340f34806e01d4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/dbabd0a9ab4c4c9f97340f34806e01d4 2023-04-19 07:55:20,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/63f2c010f22947ac891f8876aac75630 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/63f2c010f22947ac891f8876aac75630 2023-04-19 07:55:20,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/6f12d53494de412c9fcd34096d9213e3 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/6f12d53494de412c9fcd34096d9213e3 2023-04-19 07:55:20,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/54680d081a40453b845c97d0d52384f6 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/54680d081a40453b845c97d0d52384f6 2023-04-19 07:55:20,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/0fe011845276421a8b077705b1d48fcb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/0fe011845276421a8b077705b1d48fcb 2023-04-19 07:55:20,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/6b3cf74ac54d43ad8c011b20d76dd0ab to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/6b3cf74ac54d43ad8c011b20d76dd0ab 2023-04-19 07:55:20,686 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/56712b36efc4467da652b520d3e19335 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/56712b36efc4467da652b520d3e19335 2023-04-19 07:55:20,687 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/4ed88b0441eb4e72935fe9aed15a82e6 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/4ed88b0441eb4e72935fe9aed15a82e6 2023-04-19 07:55:20,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/d241f8529f694e15a778fd9129b073fe to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/d241f8529f694e15a778fd9129b073fe 2023-04-19 07:55:20,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/58243ed8782e477b8590a6575c71eebc to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/58243ed8782e477b8590a6575c71eebc 2023-04-19 07:55:20,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/db71cd384da34449b0aa70c0c612a0fb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/db71cd384da34449b0aa70c0c612a0fb 2023-04-19 07:55:20,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/efec1f880c874c52b59c4a61f4e811bb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/efec1f880c874c52b59c4a61f4e811bb 2023-04-19 07:55:20,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/1623a47b8cb943de8679ca737597fc02 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/1623a47b8cb943de8679ca737597fc02 2023-04-19 07:55:20,692 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/3346d2661310481599e57418e034bdc8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/3346d2661310481599e57418e034bdc8 2023-04-19 07:55:20,693 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/4320729861f14e79acf45a734bec4edc to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/4320729861f14e79acf45a734bec4edc 2023-04-19 07:55:20,694 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/55dbcb675b0c4c8281fe7c6b0b668b7f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/55dbcb675b0c4c8281fe7c6b0b668b7f 2023-04-19 07:55:20,694 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/76ffed16f6f345ddafc2f66f2a4fa956 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/76ffed16f6f345ddafc2f66f2a4fa956 2023-04-19 07:55:20,695 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cf33e80b900540a9a78b98ab3574d366 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cf33e80b900540a9a78b98ab3574d366 2023-04-19 07:55:20,696 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/56b67d57e4c34a08ab12711af1cc8fc7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/56b67d57e4c34a08ab12711af1cc8fc7 2023-04-19 07:55:20,697 DEBUG [StoreCloser-TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/311c0d201de14d1fbf4ca08eda03b7de to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/311c0d201de14d1fbf4ca08eda03b7de 2023-04-19 07:55:20,701 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/recovered.edits/861.seqid, newMaxSeqId=861, maxSeqId=1 2023-04-19 07:55:20,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c. 2023-04-19 07:55:20,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for cfaed3abed1fbb0d5a48467b790cb91c: 2023-04-19 07:55:20,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:20,704 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=cfaed3abed1fbb0d5a48467b790cb91c, regionState=CLOSED 2023-04-19 07:55:20,704 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890920704"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890920704"}]},"ts":"1681890920704"} 2023-04-19 07:55:20,706 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-19 07:55:20,706 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure cfaed3abed1fbb0d5a48467b790cb91c, server=jenkins-hbase4.apache.org,42359,1681890819191 in 1.5510 sec 2023-04-19 07:55:20,707 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-19 07:55:20,707 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=cfaed3abed1fbb0d5a48467b790cb91c, UNASSIGN in 1.5550 sec 2023-04-19 07:55:20,708 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890920708"}]},"ts":"1681890920708"} 2023-04-19 07:55:20,708 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-19 07:55:20,710 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-19 07:55:20,712 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5670 sec 2023-04-19 07:55:21,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-19 07:55:21,251 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-19 07:55:21,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-19 07:55:21,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:21,254 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:21,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-19 07:55:21,255 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:21,259 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:21,261 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/recovered.edits] 2023-04-19 07:55:21,264 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/3367d1bdac7f45518cdfe3902d1a7ef6 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/3367d1bdac7f45518cdfe3902d1a7ef6 2023-04-19 07:55:21,265 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/a3941249907d4b2d9c76cb9ceb9630ff to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/a3941249907d4b2d9c76cb9ceb9630ff 2023-04-19 07:55:21,266 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/a853d8bfb8094629a56221ad80046b04 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/A/a853d8bfb8094629a56221ad80046b04 2023-04-19 07:55:21,267 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/2f30121cc2954da99c4d4825772d4859 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/2f30121cc2954da99c4d4825772d4859 2023-04-19 07:55:21,268 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/70981e901cc344f7b73898e262f5567c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/70981e901cc344f7b73898e262f5567c 2023-04-19 07:55:21,269 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/b226736eafd548e4ac68b05e3a485f1d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/B/b226736eafd548e4ac68b05e3a485f1d 2023-04-19 07:55:21,271 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/8ce9f973850a4e61b2e3ed4297d20abb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/8ce9f973850a4e61b2e3ed4297d20abb 2023-04-19 07:55:21,271 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/bdb694f67056454ba12b1e37da4e4a46 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/bdb694f67056454ba12b1e37da4e4a46 2023-04-19 07:55:21,272 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cd7a3e201be84f1fbdb2264d601d0b5a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/C/cd7a3e201be84f1fbdb2264d601d0b5a 2023-04-19 07:55:21,274 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/recovered.edits/861.seqid to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c/recovered.edits/861.seqid 2023-04-19 07:55:21,275 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/cfaed3abed1fbb0d5a48467b790cb91c 2023-04-19 07:55:21,275 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 07:55:21,277 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:21,278 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-19 07:55:21,279 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-19 07:55:21,280 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:21,280 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-19 07:55:21,280 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681890921280"}]},"ts":"9223372036854775807"} 2023-04-19 07:55:21,282 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-19 07:55:21,282 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => cfaed3abed1fbb0d5a48467b790cb91c, NAME => 'TestAcidGuarantees,,1681890888107.cfaed3abed1fbb0d5a48467b790cb91c.', STARTKEY => '', ENDKEY => ''}] 2023-04-19 07:55:21,282 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-19 07:55:21,282 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681890921282"}]},"ts":"9223372036854775807"} 2023-04-19 07:55:21,283 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-19 07:55:21,285 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:21,286 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-19 07:55:21,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-19 07:55:21,356 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-19 07:55:21,367 INFO [Listener at localhost/42517] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 224), OpenFileDescriptor=514 (was 535), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=352 (was 380), ProcessCount=169 (was 169), AvailableMemoryMB=6131 (was 6257) 2023-04-19 07:55:21,377 INFO [Listener at localhost/42517] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=514, MaxFileDescriptor=60000, SystemLoadAverage=352, ProcessCount=169, AvailableMemoryMB=6131 2023-04-19 07:55:21,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 07:55:21,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 07:55:21,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:21,381 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-19 07:55:21,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-19 07:55:21,382 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-19 07:55:21,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-19 07:55:21,383 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:21,383 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31 empty. 2023-04-19 07:55:21,384 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:21,384 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 07:55:21,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-19 07:55:21,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-19 07:55:21,790 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 07:55:21,792 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a8342afdd5fe71a59bc1e541995ded31, NAME => 'TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp 2023-04-19 07:55:21,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-19 07:55:22,197 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:55:22,198 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a8342afdd5fe71a59bc1e541995ded31, disabling compactions & flushes 2023-04-19 07:55:22,198 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:22,198 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:22,198 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. after waiting 0 ms 2023-04-19 07:55:22,198 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:22,198 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:22,198 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:22,200 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-19 07:55:22,201 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890922200"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890922200"}]},"ts":"1681890922200"} 2023-04-19 07:55:22,202 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-19 07:55:22,202 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-19 07:55:22,202 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890922202"}]},"ts":"1681890922202"} 2023-04-19 07:55:22,203 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-19 07:55:22,206 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8342afdd5fe71a59bc1e541995ded31, ASSIGN}] 2023-04-19 07:55:22,208 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8342afdd5fe71a59bc1e541995ded31, ASSIGN 2023-04-19 07:55:22,208 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8342afdd5fe71a59bc1e541995ded31, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42359,1681890819191; forceNewPlan=false, retain=false 2023-04-19 07:55:22,359 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=a8342afdd5fe71a59bc1e541995ded31, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:22,360 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890922359"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890922359"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890922359"}]},"ts":"1681890922359"} 2023-04-19 07:55:22,361 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:55:22,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-19 07:55:22,515 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:22,515 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a8342afdd5fe71a59bc1e541995ded31, NAME => 'TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.', STARTKEY => '', ENDKEY => ''} 2023-04-19 07:55:22,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:22,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:55:22,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:22,516 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:22,517 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:22,518 DEBUG [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A 2023-04-19 07:55:22,518 DEBUG [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A 2023-04-19 07:55:22,518 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:55:22,519 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8342afdd5fe71a59bc1e541995ded31 columnFamilyName A 2023-04-19 07:55:22,519 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.HStore(310): Store=a8342afdd5fe71a59bc1e541995ded31/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:55:22,519 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:22,520 DEBUG [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B 2023-04-19 07:55:22,520 DEBUG [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B 2023-04-19 07:55:22,520 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:55:22,521 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8342afdd5fe71a59bc1e541995ded31 columnFamilyName B 2023-04-19 07:55:22,521 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.HStore(310): Store=a8342afdd5fe71a59bc1e541995ded31/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:55:22,521 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:22,522 DEBUG [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C 2023-04-19 07:55:22,522 DEBUG [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C 2023-04-19 07:55:22,522 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:55:22,522 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8342afdd5fe71a59bc1e541995ded31 columnFamilyName C 2023-04-19 07:55:22,523 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.HStore(310): Store=a8342afdd5fe71a59bc1e541995ded31/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:55:22,523 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:22,523 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:22,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:22,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 07:55:22,526 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:22,528 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 07:55:22,528 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a8342afdd5fe71a59bc1e541995ded31; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73048306, jitterRate=0.08850458264350891}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 07:55:22,528 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:22,529 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., pid=37, masterSystemTime=1681890922512 2023-04-19 07:55:22,530 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:22,530 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:22,530 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=a8342afdd5fe71a59bc1e541995ded31, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:22,530 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890922530"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681890922530"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681890922530"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681890922530"}]},"ts":"1681890922530"} 2023-04-19 07:55:22,532 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-19 07:55:22,532 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 in 170 msec 2023-04-19 07:55:22,533 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-19 07:55:22,533 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8342afdd5fe71a59bc1e541995ded31, ASSIGN in 326 msec 2023-04-19 07:55:22,534 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-19 07:55:22,534 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890922534"}]},"ts":"1681890922534"} 2023-04-19 07:55:22,534 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-19 07:55:22,536 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-19 07:55:22,537 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-04-19 07:55:23,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-19 07:55:23,486 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-19 07:55:23,487 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x0279325e to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:23,492 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29784e8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:23,493 DEBUG [Listener at localhost/42517] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:23,495 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46922, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:23,496 DEBUG [Listener at localhost/42517] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-19 07:55:23,497 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57146, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-19 07:55:23,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 07:55:23,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 07:55:23,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:23,908 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-19 07:55:23,909 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 07:55:23,909 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-19 07:55:23,911 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-19 07:55:23,913 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8342afdd5fe71a59bc1e541995ded31, REOPEN/MOVE}] 2023-04-19 07:55:23,914 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8342afdd5fe71a59bc1e541995ded31, REOPEN/MOVE 2023-04-19 07:55:23,914 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a8342afdd5fe71a59bc1e541995ded31, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:23,915 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890923914"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890923914"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890923914"}]},"ts":"1681890923914"} 2023-04-19 07:55:23,915 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:55:24,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:24,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a8342afdd5fe71a59bc1e541995ded31, disabling compactions & flushes 2023-04-19 07:55:24,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. after waiting 0 ms 2023-04-19 07:55:24,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-19 07:55:24,072 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:24,072 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: a8342afdd5fe71a59bc1e541995ded31 to self. 2023-04-19 07:55:24,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:24,073 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a8342afdd5fe71a59bc1e541995ded31, regionState=CLOSED 2023-04-19 07:55:24,074 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890924073"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890924073"}]},"ts":"1681890924073"} 2023-04-19 07:55:24,076 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-19 07:55:24,076 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 in 159 msec 2023-04-19 07:55:24,076 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8342afdd5fe71a59bc1e541995ded31, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,42359,1681890819191; forceNewPlan=false, retain=true 2023-04-19 07:55:24,226 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a8342afdd5fe71a59bc1e541995ded31, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,227 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890924226"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890924226"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890924226"}]},"ts":"1681890924226"} 2023-04-19 07:55:24,228 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:55:24,382 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,382 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a8342afdd5fe71a59bc1e541995ded31, NAME => 'TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.', STARTKEY => '', ENDKEY => ''} 2023-04-19 07:55:24,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:24,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:55:24,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:24,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:24,384 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:24,385 DEBUG [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A 2023-04-19 07:55:24,385 DEBUG [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A 2023-04-19 07:55:24,385 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:55:24,386 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8342afdd5fe71a59bc1e541995ded31 columnFamilyName A 2023-04-19 07:55:24,387 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.HStore(310): Store=a8342afdd5fe71a59bc1e541995ded31/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:55:24,387 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:24,387 DEBUG [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B 2023-04-19 07:55:24,388 DEBUG [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B 2023-04-19 07:55:24,388 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:55:24,388 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8342afdd5fe71a59bc1e541995ded31 columnFamilyName B 2023-04-19 07:55:24,388 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.HStore(310): Store=a8342afdd5fe71a59bc1e541995ded31/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:55:24,388 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:24,389 DEBUG [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C 2023-04-19 07:55:24,389 DEBUG [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C 2023-04-19 07:55:24,389 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:55:24,389 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8342afdd5fe71a59bc1e541995ded31 columnFamilyName C 2023-04-19 07:55:24,389 INFO [StoreOpener-a8342afdd5fe71a59bc1e541995ded31-1] regionserver.HStore(310): Store=a8342afdd5fe71a59bc1e541995ded31/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:55:24,390 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:24,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:24,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 07:55:24,394 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:24,394 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a8342afdd5fe71a59bc1e541995ded31; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69552577, jitterRate=0.03641416132450104}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 07:55:24,394 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:24,395 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., pid=42, masterSystemTime=1681890924379 2023-04-19 07:55:24,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,396 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a8342afdd5fe71a59bc1e541995ded31, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,396 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890924396"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681890924396"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681890924396"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681890924396"}]},"ts":"1681890924396"} 2023-04-19 07:55:24,398 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-19 07:55:24,398 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 in 169 msec 2023-04-19 07:55:24,399 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-19 07:55:24,399 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8342afdd5fe71a59bc1e541995ded31, REOPEN/MOVE in 485 msec 2023-04-19 07:55:24,400 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-19 07:55:24,400 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-19 07:55:24,402 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 902 msec 2023-04-19 07:55:24,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-19 07:55:24,403 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x4473f45c to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:24,407 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fd3d54e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:24,407 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x41507d16 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:24,414 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a5ab074, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:24,414 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x49b5bd0d to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:24,418 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bcfc1e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:24,419 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x5c8d927d to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:24,422 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38341134, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:24,422 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x419b69bf to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:24,425 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71b5058b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:24,426 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x3620ee35 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:24,430 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f603c27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:24,430 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:24,433 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:24,434 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x10377b50 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:24,439 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ccd8b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:24,439 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:24,443 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:24,443 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x6a35a28d to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:24,447 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ec12b02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:24,453 DEBUG [hconnection-0x36e7bef7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:24,453 DEBUG [hconnection-0x676a4807-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:24,453 DEBUG [hconnection-0x7ea8fa00-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:24,454 DEBUG [hconnection-0x2e1591e7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:24,454 DEBUG [hconnection-0x1ac87934-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:24,454 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46936, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:24,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:24,455 DEBUG [hconnection-0x7925fd91-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:24,455 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46972, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:24,455 DEBUG [hconnection-0x2757fa82-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:24,458 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46958, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:24,458 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46946, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:24,458 DEBUG [hconnection-0x2758d2a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:24,461 DEBUG [hconnection-0x4c930219-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:24,461 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46978, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:24,460 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46996, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:24,459 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46994, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:24,459 DEBUG [hconnection-0xb565163-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:24,463 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47022, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:24,464 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47008, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:24,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:24,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:24,465 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:24,465 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:24,467 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:24,467 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47030, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:24,467 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:24,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,472 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,472 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:24,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:24,472 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,472 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:24,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:24,476 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:55:24,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:24,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:24,478 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,478 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:24,478 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:24,478 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:24,478 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:24,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:24,479 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:24,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:24,479 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:24,479 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:24,483 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,484 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,484 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:24,484 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:24,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:24,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1295d426[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:24,486 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:24,486 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,488 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,488 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:24,488 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:24,488 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:24,488 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,488 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:24,491 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:24,491 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:24,491 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,491 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:24,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:24,493 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:24,493 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:24,493 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:24,493 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:24,493 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:24,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:24,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:24,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:24,496 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,497 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,499 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,499 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:24,499 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:24,499 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,499 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:24,499 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:24,499 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,500 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:24,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:24,500 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,500 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,500 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:24,500 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:24,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:24,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:24,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681890924487, max timestamp=1681890924513Num uniques 859; 2023-04-19 07:55:24,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:55:24,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681890924487, max timestamp=1681890924513Num uniques 859; 2023-04-19 07:55:24,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:55:24,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681890924487, max timestamp=1681890924513Num uniques 859; 2023-04-19 07:55:24,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:55:24,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890984529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47030 deadline: 1681890984530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890984530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890984534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890984536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,600 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:24,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:24,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:24,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:24,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:24,604 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:24,604 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:24,604 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:24,604 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:24,604 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,604 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,606 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,606 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:24,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:24,606 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,606 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:24,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:24,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:24,608 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:24,608 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:24,608 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:24,612 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,612 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:24,612 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:24,612 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:24,612 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,615 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:24,615 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:24,618 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,618 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:24,618 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,618 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:24,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:24,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57e6b478[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:24,619 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:24,619 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,621 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:24,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:24,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:24,622 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,622 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:24,625 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,625 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:24,625 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,625 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:24,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:24,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:24,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:24,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:24,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:24,628 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:24,628 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:24,628 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:24,628 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:24,628 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:24,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:24,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:24,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:24,636 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,638 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:24,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47030 deadline: 1681890984638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890984638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890984641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890984643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890984643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,649 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,649 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,649 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,649 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:24,649 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:24,649 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:24,649 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:24,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:24,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:24,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:24,649 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,650 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,650 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:24,650 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:24,650 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:24,750 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:24,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:24,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:24,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:24,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:24,768 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:24,768 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:24,768 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:24,768 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:24,769 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,769 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,770 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,770 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:24,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:24,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:24,771 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,771 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:24,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:24,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:24,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:24,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:24,782 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:24,782 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:24,782 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:24,784 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,784 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:24,784 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:24,784 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,784 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:24,785 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:24,786 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:24,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,789 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,789 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:24,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1283e5c2[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:24,790 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:24,790 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,794 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,794 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,794 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:24,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:24,796 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:24,796 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:24,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:24,800 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:24,800 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:24,800 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:24,800 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:24,800 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:24,801 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,801 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:24,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,802 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:24,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:24,802 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,803 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,806 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,806 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:24,806 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:24,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:24,806 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,806 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,806 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,806 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:24,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:24,806 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:24,806 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,806 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:24,806 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:24,807 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:24,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47030 deadline: 1681890984841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890984842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890984842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890984844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:24,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890984844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,900 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419f0fc162207294263b4de67df1339a684 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f0fc162207294263b4de67df1339a684 2023-04-19 07:55:24,907 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:24,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:24,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:24,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:24,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:24,914 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:24,914 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:24,914 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:24,914 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:24,914 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,914 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,919 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,919 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:24,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:24,920 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,921 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:24,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:24,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:24,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:24,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:24,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:24,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:24,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:24,923 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,923 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:24,923 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:24,923 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:24,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:24,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:24,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:24,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:24,926 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:24,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:24,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b6c49f4[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:24,928 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:24,928 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,929 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,929 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:24,929 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:24,929 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:24,929 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,929 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:24,932 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:24,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:24,934 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:24,935 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:24,935 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:24,935 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:24,935 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:24,937 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:24,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:24,938 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,940 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,940 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:24,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,940 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:24,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:24,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:24,944 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:24,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:24,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:24,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:24,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:24,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:24,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:24,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:24,944 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:24,944 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:24,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,044 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:25,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:25,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:25,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:25,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:25,052 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:25,052 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,052 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:25,052 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:25,053 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,053 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,054 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,054 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,055 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,055 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:25,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:25,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:25,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:25,063 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:25,063 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,063 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:25,064 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:25,064 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:25,064 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:25,064 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:25,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:25,067 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:25,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6764925b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:25,069 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:25,069 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,071 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,072 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:25,072 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:25,072 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:25,072 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,072 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:25,074 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:25,074 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:25,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,074 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:25,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:25,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:25,077 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:25,077 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:25,077 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:25,078 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:25,078 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:25,079 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,080 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,080 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,081 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:25,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,081 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,088 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,088 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,089 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,089 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,089 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:25,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:25,089 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:25,089 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,089 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:25,089 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,089 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,089 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:25,089 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:25,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:25,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:25,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890985149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:25,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47030 deadline: 1681890985149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:25,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890985149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:25,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890985150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:25,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890985150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,189 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:25,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:25,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:25,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:25,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:25,196 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:25,196 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,196 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:25,196 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:25,197 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,197 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,198 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,198 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,198 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,198 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:25,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:25,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:25,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:25,203 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:25,203 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,203 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:25,204 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:25,204 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:25,204 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:25,204 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:25,206 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,206 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,206 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,206 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:25,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@543afbb[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:25,208 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:25,208 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,211 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,211 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:25,211 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:25,211 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:25,211 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,211 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:25,212 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,212 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,213 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:25,213 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:25,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:25,215 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:25,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:25,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:25,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:25,215 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:25,218 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,218 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,218 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,219 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,221 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,224 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,224 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,224 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:25,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,224 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:25,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:25,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:25,224 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,224 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:25,225 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:25,224 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,225 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:25,225 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,225 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,225 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,314 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/edd34196153246fd897db9e892abbbf3 2023-04-19 07:55:25,325 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:25,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:25,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:25,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:25,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:25,332 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:25,332 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,332 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:25,332 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:25,332 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,333 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,335 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,335 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,335 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,335 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:25,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:25,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:25,340 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:25,340 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:25,340 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:25,340 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:25,341 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:25,342 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,342 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,342 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,342 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:25,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:25,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@863f724[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:25,344 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:25,344 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,345 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,345 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:25,345 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,345 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:25,345 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:25,345 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:25,347 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,347 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:25,347 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,347 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:25,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:25,350 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:25,350 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:25,350 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:25,350 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:25,350 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:25,352 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,353 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,353 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,353 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:25,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,354 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,358 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,358 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,358 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,358 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:25,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,358 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:25,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:25,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:25,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:25,358 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,359 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,359 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,359 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,359 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,359 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:25,359 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:25,459 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:25,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:25,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:25,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:25,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:25,462 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:25,462 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,462 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:25,462 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:25,463 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,463 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,464 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,464 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,465 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,465 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:25,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:25,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:25,466 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:25,466 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:25,466 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:25,466 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:25,468 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,468 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,468 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,468 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:25,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31792c35[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:25,470 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:25,470 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,472 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,472 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:25,472 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:25,472 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,473 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:25,472 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:25,474 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:25,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:25,476 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:25,477 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:25,477 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:25,477 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:25,477 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:25,479 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,479 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,479 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:25,479 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,480 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,486 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,486 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:25,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,486 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,486 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:25,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:25,486 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,487 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,487 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,487 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,487 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,487 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,487 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:25,487 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:25,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:25,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,587 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:25,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:25,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:25,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:25,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:25,590 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:25,590 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,590 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:25,591 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:25,591 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,591 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,592 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,593 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,593 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:25,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:25,594 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:25,594 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,594 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:25,594 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:25,594 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:25,594 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:25,594 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:25,594 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:25,594 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:25,594 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:25,594 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:25,594 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,594 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:25,594 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:25,596 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,596 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,596 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,596 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:25,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72cc860b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:25,598 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:25,598 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,599 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:25,600 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,600 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:25,601 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:25,601 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:25,601 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,601 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:25,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:25,603 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:25,604 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:25,604 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:25,604 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:25,604 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:25,605 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,605 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,605 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:25,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,606 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,607 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,609 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,609 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,609 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,609 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,609 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,609 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,610 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:25,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,610 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:25,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:25,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:25,610 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,610 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,610 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:25,610 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:25,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:25,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890985656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:25,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890985656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:25,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890985658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:25,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:25,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890985658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47030 deadline: 1681890985659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,710 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:25,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:25,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:25,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:25,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:25,716 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:25,716 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,716 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:25,716 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:25,716 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,717 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,719 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,719 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,720 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,720 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:25,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:25,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:25,721 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:25,721 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,721 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:25,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:25,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:25,724 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:25,724 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:25,724 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:25,724 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:25,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:25,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:25,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:25,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:25,726 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,726 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/0051e3e1dd9e453d869cba288b28023f 2023-04-19 07:55:25,727 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,727 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:25,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aead19e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:25,730 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:25,730 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,732 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,732 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:25,732 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:25,732 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:25,732 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,732 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:25,734 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:25,734 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:25,734 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,734 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:25,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:25,739 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:25,739 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:25,739 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:25,739 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:25,739 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:25,745 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,745 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,745 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,746 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,748 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:25,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,750 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,751 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,751 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:25,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:25,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:25,751 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:25,751 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:25,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:25,851 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:25,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:25,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:25,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:25,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:25,855 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:25,855 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,855 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:25,855 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:25,855 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,855 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,857 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,857 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,857 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,857 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:25,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:25,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:25,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:25,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:25,859 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:25,859 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,859 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:25,859 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:25,859 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:25,859 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:25,859 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:25,859 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:25,859 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:25,859 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:25,859 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:25,859 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:25,861 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,861 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,862 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,862 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:25,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:25,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7da7c130[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:25,864 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:25,864 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,871 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,871 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:25,871 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:25,871 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:25,871 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,871 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:25,873 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,873 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:25,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:25,875 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:25,875 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:25,875 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:25,875 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:25,875 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:25,877 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,877 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,877 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:25,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,877 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:25,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,878 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:25,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:25,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,880 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,880 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,880 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:25,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,880 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:25,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:25,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:25,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:25,880 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:25,881 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:25,881 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:25,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:25,881 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,881 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,881 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,881 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:25,881 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,981 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:25,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:25,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:25,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:25,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:25,985 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:25,985 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:25,985 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:25,985 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:25,985 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,985 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,991 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:25,991 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,992 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:25,992 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:25,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:25,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:25,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:26,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:26,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:26,008 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:26,016 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:26,016 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:26,016 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:26,016 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:26,016 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:26,016 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:26,016 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:26,016 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:26,016 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,016 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:26,016 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:26,018 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,018 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:26,018 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,018 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:26,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:26,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c75235[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:26,020 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:26,020 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,021 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:26,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:26,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:26,021 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,021 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:26,024 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:26,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:26,026 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:26,026 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:26,026 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:26,026 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:26,026 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:26,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:26,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:26,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:26,029 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,030 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:26,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:26,032 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:26,032 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:26,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:26,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:26,033 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:26,133 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:26,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:26,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:26,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:26,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:26,137 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:26,137 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:26,137 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:26,137 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:26,138 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,138 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,139 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:26,139 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:26,139 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,139 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:26,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:26,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:26,144 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:26,144 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:26,144 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:26,144 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:26,144 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:26,144 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:26,144 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:26,145 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:26,145 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:26,145 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:26,147 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,147 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:26,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:26,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13ad7058[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:26,149 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:26,149 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,150 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:26,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:26,151 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:26,151 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,151 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:26,152 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,152 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:26,152 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:26,152 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:26,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:26,155 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:26,155 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:26,155 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:26,155 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:26,155 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:26,157 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,157 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:26,157 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:26,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:26,157 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,159 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:26,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:26,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:26,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,163 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:26,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:26,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:26,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:26,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:26,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,163 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:26,163 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:26,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:26,166 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/00c5cd1086564da99c59becbc657bcd9 2023-04-19 07:55:26,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/edd34196153246fd897db9e892abbbf3 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/edd34196153246fd897db9e892abbbf3 2023-04-19 07:55:26,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/edd34196153246fd897db9e892abbbf3, entries=250, sequenceid=16, filesize=38.7 K 2023-04-19 07:55:26,176 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/0051e3e1dd9e453d869cba288b28023f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/0051e3e1dd9e453d869cba288b28023f 2023-04-19 07:55:26,179 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/0051e3e1dd9e453d869cba288b28023f, entries=150, sequenceid=16, filesize=11.6 K 2023-04-19 07:55:26,180 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/00c5cd1086564da99c59becbc657bcd9 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/00c5cd1086564da99c59becbc657bcd9 2023-04-19 07:55:26,184 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/00c5cd1086564da99c59becbc657bcd9, entries=150, sequenceid=16, filesize=11.6 K 2023-04-19 07:55:26,185 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for a8342afdd5fe71a59bc1e541995ded31 in 1709ms, sequenceid=16, compaction requested=false 2023-04-19 07:55:26,185 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:26,240 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-19 07:55:26,263 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:26,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:26,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:26,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:26,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:26,268 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:26,268 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:26,268 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:26,268 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:26,268 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,268 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,270 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:26,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:26,270 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,270 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:26,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:26,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:26,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:26,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:26,274 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:26,274 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:26,274 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:26,275 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:26,275 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:26,275 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:26,275 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:26,275 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-19 07:55:26,276 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:26,276 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:26,276 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:26,276 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:26,276 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:26,276 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:26,294 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419d2db9c243d61454c8fba6771d8434907 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d2db9c243d61454c8fba6771d8434907 2023-04-19 07:55:26,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:26,661 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:26,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:26,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890986674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:26,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47030 deadline: 1681890986675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:26,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890986681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:26,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890986682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:26,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890986682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,715 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/85d94892b2014b978e44718c02987121 2023-04-19 07:55:26,732 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/f3aae6c9714d4616a78b0e3a38938b71 2023-04-19 07:55:26,749 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/33e8e1fc2921498f9aa6cfdb37c81047 2023-04-19 07:55:26,753 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/85d94892b2014b978e44718c02987121 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/85d94892b2014b978e44718c02987121 2023-04-19 07:55:26,758 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/85d94892b2014b978e44718c02987121, entries=150, sequenceid=50, filesize=25.1 K 2023-04-19 07:55:26,759 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/f3aae6c9714d4616a78b0e3a38938b71 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f3aae6c9714d4616a78b0e3a38938b71 2023-04-19 07:55:26,763 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f3aae6c9714d4616a78b0e3a38938b71, entries=150, sequenceid=50, filesize=11.6 K 2023-04-19 07:55:26,764 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/33e8e1fc2921498f9aa6cfdb37c81047 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/33e8e1fc2921498f9aa6cfdb37c81047 2023-04-19 07:55:26,767 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/33e8e1fc2921498f9aa6cfdb37c81047, entries=150, sequenceid=50, filesize=11.6 K 2023-04-19 07:55:26,768 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for a8342afdd5fe71a59bc1e541995ded31 in 492ms, sequenceid=50, compaction requested=false 2023-04-19 07:55:26,768 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:26,768 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:26,770 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,770 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:26,770 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:26,771 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:26,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:26,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f772592[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:26,772 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:26,772 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,773 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,774 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:26,774 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:26,774 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:26,774 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,774 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:26,776 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:26,776 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:26,776 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,776 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:26,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:26,780 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:26,780 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:26,780 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:26,780 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:26,780 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:26,782 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:26,782 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,783 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:26,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:26,784 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,785 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,787 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:26,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:26,787 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-19 07:55:26,787 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,787 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,787 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,787 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:26,787 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,788 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:26,788 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:26,788 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:26,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:26,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:26,788 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,788 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:26,788 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:26,788 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:26,788 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:26,788 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:26,788 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:26,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:26,788 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:26,788 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:26,819 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419b7897e49cb4a462abc8000953dd15637 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b7897e49cb4a462abc8000953dd15637 2023-04-19 07:55:26,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:26,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890986819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:26,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890986824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:26,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890986825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:26,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890986828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,837 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/88e90b35fb5247c78eca522a3d2dfbaf 2023-04-19 07:55:26,889 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:26,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:26,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:26,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:26,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:26,892 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:26,892 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:26,892 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:26,892 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:26,892 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,892 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,894 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,894 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:26,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:26,894 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,894 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:26,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:26,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:26,895 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:26,895 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:26,895 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:26,895 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:26,898 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,898 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:26,898 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:26,898 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:26,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:26,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c9acf0d[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:26,900 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:26,900 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,905 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,905 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:26,905 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:26,905 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:26,906 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,906 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:26,907 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:26,907 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,907 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,907 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:26,909 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:26,909 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:26,909 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:26,909 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:26,909 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:26,911 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,911 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,911 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:26,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:26,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:26,911 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,913 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:26,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:26,915 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:26,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:26,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:26,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:26,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:26,915 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:26,915 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:26,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:26,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:26,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:26,915 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:26,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:26,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890986929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:26,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890986929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:26,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890986935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:26,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:26,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890986935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,015 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:27,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:27,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:27,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:27,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:27,019 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:27,019 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:27,019 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:27,019 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:27,019 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,019 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,021 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:27,021 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:27,022 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,022 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:27,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:27,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:27,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:27,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:27,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:27,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:27,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:27,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:27,023 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,023 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:27,023 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:27,024 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:27,025 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,026 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,026 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53f36e51[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:27,027 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:27,027 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,028 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:27,028 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:27,028 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,029 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:27,029 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:27,030 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,030 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:27,030 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,031 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:27,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:27,033 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:27,033 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:27,033 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:27,033 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:27,033 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:27,036 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,036 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,036 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:27,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:27,036 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,037 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:27,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:27,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,040 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:27,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:27,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:27,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:27,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:27,040 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:27,040 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:27,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:27,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890987134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:27,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890987134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,140 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:27,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:27,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:27,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:27,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890987140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:27,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890987141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:27,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:27,145 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:27,145 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:27,145 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:27,145 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:27,145 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,145 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,147 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,147 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:27,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:27,147 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,147 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:27,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:27,149 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:27,150 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,150 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:27,150 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:27,150 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:27,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:27,151 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,151 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:27,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b24fbc7[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:27,153 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:27,153 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,155 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,155 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:27,155 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:27,155 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:27,155 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,156 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:27,157 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,157 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:27,157 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,157 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:27,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:27,160 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:27,160 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:27,160 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:27,160 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:27,160 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:27,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:27,164 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,165 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,167 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:27,167 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:27,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:27,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:27,167 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:27,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:27,168 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:27,168 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:27,167 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,168 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:27,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/c968cfcfd986498d916e4173dd0a4cb5 2023-04-19 07:55:27,270 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:27,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:27,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:27,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:27,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:27,274 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:27,274 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:27,274 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:27,274 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:27,274 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,274 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,276 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,276 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:27,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:27,276 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,276 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:27,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:27,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:27,277 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,278 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:27,278 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:27,278 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,278 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:27,278 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:27,278 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,278 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:27,278 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,279 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:27,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3de90084[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:27,281 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:27,281 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,283 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,283 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:27,283 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:27,283 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:27,283 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,283 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:27,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,287 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,287 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:27,287 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:27,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:27,289 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:27,289 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:27,289 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:27,289 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:27,289 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:27,290 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,290 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,290 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,293 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,294 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,296 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,296 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,296 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:27,296 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:27,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:27,296 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,296 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:27,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:27,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:27,296 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:27,296 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,296 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,296 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:27,296 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,296 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:27,397 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:27,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:27,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:27,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:27,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:27,401 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:27,401 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:27,401 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:27,401 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:27,402 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,402 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,403 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:27,403 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:27,404 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,404 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:27,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:27,405 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,405 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:27,405 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:27,405 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,406 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:27,406 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:27,408 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,408 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,408 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,408 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:27,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b14293[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:27,410 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:27,410 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,411 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:27,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:27,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:27,412 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,412 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:27,413 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:27,413 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,413 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,413 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:27,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:27,415 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:27,416 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:27,416 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:27,416 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:27,416 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:27,417 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,417 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,417 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:27,417 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,419 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,422 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,422 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,422 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,422 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:27,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:27,422 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:27,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:27,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:27,422 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,422 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:27,422 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:27,422 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,422 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:27,422 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,422 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:27,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:27,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:27,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890987440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:27,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890987440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:27,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890987446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:27,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890987446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,523 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:27,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:27,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:27,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:27,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:27,530 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:27,530 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:27,530 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:27,530 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:27,531 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,531 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,532 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:27,532 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:27,532 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,532 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:27,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:27,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:27,533 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:27,533 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:27,533 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:27,534 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,534 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:27,534 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:27,534 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:27,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:27,535 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:27,537 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:27,537 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ff835d2[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:27,538 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,538 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,538 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:27,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,539 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:27,539 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:27,539 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:27,545 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,545 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:27,545 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,545 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:27,548 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:27,548 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:27,548 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:27,548 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:27,548 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:27,549 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,549 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,549 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:27,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:27,551 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,552 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,554 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:27,554 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,554 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,554 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,554 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,554 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,554 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,554 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,554 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:27,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:27,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:27,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:27,555 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:27,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:27,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:27,556 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:27,556 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:27,656 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:27,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:27,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:27,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:27,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:27,659 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:27,659 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:27,659 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:27,659 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:27,660 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,660 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,662 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:27,662 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:27,662 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,662 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:27,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:27,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:27,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:27,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:27,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:27,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:27,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:27,664 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,664 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:27,664 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:27,664 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:27,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:27,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:27,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:27,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,669 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,669 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/3104641bf90a40d5a697a797227013c4 2023-04-19 07:55:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:27,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@438e0529[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:27,671 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:27,671 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:27,673 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,673 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:27,674 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:27,674 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,675 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/88e90b35fb5247c78eca522a3d2dfbaf as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/88e90b35fb5247c78eca522a3d2dfbaf 2023-04-19 07:55:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/88e90b35fb5247c78eca522a3d2dfbaf, entries=200, sequenceid=63, filesize=31.9 K 2023-04-19 07:55:27,679 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/c968cfcfd986498d916e4173dd0a4cb5 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/c968cfcfd986498d916e4173dd0a4cb5 2023-04-19 07:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:27,680 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:27,680 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:27,680 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:27,680 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:27,680 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:27,682 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,682 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:27,682 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:27,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:27,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:27,682 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/c968cfcfd986498d916e4173dd0a4cb5, entries=150, sequenceid=63, filesize=11.6 K 2023-04-19 07:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,684 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/3104641bf90a40d5a697a797227013c4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3104641bf90a40d5a697a797227013c4 2023-04-19 07:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3104641bf90a40d5a697a797227013c4, entries=150, sequenceid=63, filesize=11.6 K 2023-04-19 07:55:27,689 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for a8342afdd5fe71a59bc1e541995ded31 in 902ms, sequenceid=63, compaction requested=true 2023-04-19 07:55:27,689 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:27,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:27,689 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:27,689 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:27,690 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,689 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:27,690 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:27,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:27,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:27,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:27,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:27,690 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:27,690 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:27,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:27,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:27,690 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,690 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,690 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,690 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:27,690 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,690 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:27,690 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:27,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:27,691 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98062 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:27,691 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:27,691 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/A is initiating minor compaction (all files) 2023-04-19 07:55:27,691 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/A in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,691 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/edd34196153246fd897db9e892abbbf3, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/85d94892b2014b978e44718c02987121, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/88e90b35fb5247c78eca522a3d2dfbaf] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=95.8 K 2023-04-19 07:55:27,691 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/B is initiating minor compaction (all files) 2023-04-19 07:55:27,691 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/B in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,691 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/0051e3e1dd9e453d869cba288b28023f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f3aae6c9714d4616a78b0e3a38938b71, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/c968cfcfd986498d916e4173dd0a4cb5] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=34.8 K 2023-04-19 07:55:27,692 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting edd34196153246fd897db9e892abbbf3, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681890924470 2023-04-19 07:55:27,692 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 0051e3e1dd9e453d869cba288b28023f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681890924472 2023-04-19 07:55:27,692 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 85d94892b2014b978e44718c02987121, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681890924529 2023-04-19 07:55:27,692 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting f3aae6c9714d4616a78b0e3a38938b71, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681890924529 2023-04-19 07:55:27,693 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 88e90b35fb5247c78eca522a3d2dfbaf, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681890926679 2023-04-19 07:55:27,693 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting c968cfcfd986498d916e4173dd0a4cb5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681890926679 2023-04-19 07:55:27,699 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#B#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:27,705 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#A#compaction#307 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:27,791 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:27,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:27,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:27,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:27,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:27,794 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:27,794 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:27,794 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:27,794 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:27,795 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,795 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,796 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:27,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:27,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:27,796 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,797 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,797 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:27,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:27,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:27,798 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:27,798 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:27,798 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-19 07:55:27,798 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:27,799 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:27,799 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:27,799 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:27,799 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:27,799 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:27,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:27,952 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:27,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:27,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890987964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:27,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890987964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:27,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890987965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:27,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:27,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890987971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:28,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:28,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890988076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:28,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:28,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890988076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:28,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:28,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890988077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:28,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:28,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890988078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:28,114 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/2acafd61e3c84f3a9a7879409d57fc4b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2acafd61e3c84f3a9a7879409d57fc4b 2023-04-19 07:55:28,120 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/B of a8342afdd5fe71a59bc1e541995ded31 into 2acafd61e3c84f3a9a7879409d57fc4b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:28,120 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:28,120 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/B, priority=13, startTime=1681890927689; duration=0sec 2023-04-19 07:55:28,120 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:28,120 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:28,121 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:28,121 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/C is initiating minor compaction (all files) 2023-04-19 07:55:28,121 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/C in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:28,121 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/00c5cd1086564da99c59becbc657bcd9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/33e8e1fc2921498f9aa6cfdb37c81047, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3104641bf90a40d5a697a797227013c4] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=34.8 K 2023-04-19 07:55:28,121 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 00c5cd1086564da99c59becbc657bcd9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681890924472 2023-04-19 07:55:28,122 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 33e8e1fc2921498f9aa6cfdb37c81047, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681890924529 2023-04-19 07:55:28,122 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 3104641bf90a40d5a697a797227013c4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681890926679 2023-04-19 07:55:28,128 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#C#compaction#309 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:28,138 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/baf6e8e767804ef4b1de2cada7444128 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/baf6e8e767804ef4b1de2cada7444128 2023-04-19 07:55:28,142 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/C of a8342afdd5fe71a59bc1e541995ded31 into baf6e8e767804ef4b1de2cada7444128(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:28,142 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:28,142 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/C, priority=13, startTime=1681890927690; duration=0sec 2023-04-19 07:55:28,142 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:28,213 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304190bc9edd8a4d94a9882042c2cd2e0669b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304190bc9edd8a4d94a9882042c2cd2e0669b 2023-04-19 07:55:28,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:28,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890988283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:28,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:28,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890988283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:28,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:28,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890988283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:28,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:28,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890988284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:28,536 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/c4401770e22a4a0bbea48ebb6c7cd76a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/c4401770e22a4a0bbea48ebb6c7cd76a 2023-04-19 07:55:28,542 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/A of a8342afdd5fe71a59bc1e541995ded31 into c4401770e22a4a0bbea48ebb6c7cd76a(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:28,542 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:28,542 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/A, priority=13, startTime=1681890927689; duration=0sec 2023-04-19 07:55:28,543 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:28,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:28,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890988590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:28,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:28,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890988590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:28,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:28,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890988591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:28,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:28,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890988591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:28,620 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/0cab1c7be0294f568e6326530570a641 2023-04-19 07:55:28,642 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/141b751147c646339dbac5f178132572 2023-04-19 07:55:28,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:28,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47030 deadline: 1681890988708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:28,715 DEBUG [Thread-2221] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:55:29,074 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/79d6d41e177b4386a1b307f29ce94804 2023-04-19 07:55:29,079 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/0cab1c7be0294f568e6326530570a641 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0cab1c7be0294f568e6326530570a641 2023-04-19 07:55:29,083 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0cab1c7be0294f568e6326530570a641, entries=150, sequenceid=82, filesize=25.1 K 2023-04-19 07:55:29,084 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/141b751147c646339dbac5f178132572 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/141b751147c646339dbac5f178132572 2023-04-19 07:55:29,091 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/141b751147c646339dbac5f178132572, entries=150, sequenceid=82, filesize=11.6 K 2023-04-19 07:55:29,091 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/79d6d41e177b4386a1b307f29ce94804 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/79d6d41e177b4386a1b307f29ce94804 2023-04-19 07:55:29,095 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/79d6d41e177b4386a1b307f29ce94804, entries=150, sequenceid=82, filesize=11.6 K 2023-04-19 07:55:29,095 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for a8342afdd5fe71a59bc1e541995ded31 in 1297ms, sequenceid=82, compaction requested=false 2023-04-19 07:55:29,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:29,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:29,096 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:29,096 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:29,096 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,096 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:29,096 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:29,097 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,097 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:29,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:29,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:29,098 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,098 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:29,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:29,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:29,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3616d0fc[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:29,100 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:29,100 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,102 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,102 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:29,102 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:29,102 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:29,102 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,103 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:29,104 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:29,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:29,104 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-19 07:55:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,105 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:29,105 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:29,105 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:29,105 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:29,105 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,105 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:29,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:29,113 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:29,113 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:29,113 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:29,113 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:29,113 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:29,115 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:29,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:29,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:29,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:29,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,119 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,120 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:29,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,123 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,123 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,123 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:29,123 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:29,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:29,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:29,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:29,123 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:29,123 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:29,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:29,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:29,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890989124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890989131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890989131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,133 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419f8ad4902460d4b599eb1118bf9944e68 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f8ad4902460d4b599eb1118bf9944e68 2023-04-19 07:55:29,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890989132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,223 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:29,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:29,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:29,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:29,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:29,227 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:29,227 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:29,227 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:29,227 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:29,228 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,228 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,230 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,230 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:29,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:29,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:29,232 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,232 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:29,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:29,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:29,233 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:29,233 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:29,233 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:29,233 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:29,233 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:29,233 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:29,233 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:29,233 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:29,233 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,233 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:29,233 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:29,235 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:29,235 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:29,236 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,236 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:29,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:29,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:29,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:29,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bbeb977[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:29,237 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:29,237 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,238 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,239 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:29,239 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:29,239 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:29,239 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,239 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:29,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890989233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890989234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890989234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,241 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,241 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:29,241 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:29,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:29,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:29,241 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:29,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:29,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:29,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:29,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:29,245 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:29,245 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:29,245 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:29,245 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:29,245 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:29,247 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,247 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:29,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,247 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:29,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:29,247 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:29,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890989242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:29,250 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,251 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,251 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,251 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,251 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:29,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:29,251 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:29,251 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:29,252 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:29,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:29,251 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:29,252 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:29,252 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,252 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:29,252 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:29,352 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:29,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:29,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:29,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:29,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:29,357 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:29,357 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:29,357 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:29,357 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:29,357 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,357 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,359 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:29,359 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,359 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,359 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:29,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:29,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:29,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:29,361 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:29,361 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:29,361 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:29,365 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:29,365 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:29,365 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:29,365 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:29,365 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:29,366 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:29,367 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:29,367 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:29,367 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,367 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:29,367 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:29,369 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,369 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:29,370 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,370 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:29,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:29,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:29,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35665d77[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:29,372 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:29,373 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,374 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,374 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:29,374 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:29,374 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:29,374 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,374 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:29,375 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:29,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:29,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:29,378 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:29,378 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:29,378 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:29,378 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:29,378 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:29,380 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,380 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:29,380 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:29,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:29,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:29,381 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:29,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,382 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,389 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,389 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:29,389 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:29,389 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:29,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:29,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:29,389 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:29,389 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:29,389 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,389 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,389 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:29,389 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:29,389 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:29,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890989441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890989441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890989442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890989450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,490 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:29,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:29,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:29,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:29,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:29,492 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:29,492 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:29,492 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:29,492 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:29,493 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,493 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,494 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,494 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:29,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:29,494 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,494 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:29,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:29,495 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:29,495 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:29,495 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:29,496 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:29,496 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:29,496 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:29,496 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:29,496 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:29,496 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:29,496 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:29,496 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:29,496 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,496 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:29,496 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:29,504 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:29,504 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:29,504 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,504 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:29,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:29,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@672b1bf[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:29,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:29,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,507 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,507 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:29,507 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:29,507 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:29,507 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,507 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:29,509 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:29,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:29,511 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:29,511 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:29,511 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:29,511 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:29,511 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:29,513 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,513 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,513 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:29,513 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:29,514 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,522 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:29,522 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:29,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:29,522 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,522 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:29,522 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:29,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:29,522 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:29,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:29,522 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,522 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:29,522 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,523 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,523 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:29,523 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:29,542 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=101, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/1514a2d9cf554ad68991c8ace074310f 2023-04-19 07:55:29,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/fac58c4fa0e847978c94cb15134b7381 2023-04-19 07:55:29,604 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/652aec2a0e1a4b3ba9e0d3eb3e8f778b 2023-04-19 07:55:29,608 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/1514a2d9cf554ad68991c8ace074310f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/1514a2d9cf554ad68991c8ace074310f 2023-04-19 07:55:29,613 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/1514a2d9cf554ad68991c8ace074310f, entries=300, sequenceid=101, filesize=45.5 K 2023-04-19 07:55:29,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/fac58c4fa0e847978c94cb15134b7381 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/fac58c4fa0e847978c94cb15134b7381 2023-04-19 07:55:29,618 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/fac58c4fa0e847978c94cb15134b7381, entries=150, sequenceid=101, filesize=11.6 K 2023-04-19 07:55:29,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/652aec2a0e1a4b3ba9e0d3eb3e8f778b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/652aec2a0e1a4b3ba9e0d3eb3e8f778b 2023-04-19 07:55:29,623 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:29,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/652aec2a0e1a4b3ba9e0d3eb3e8f778b, entries=150, sequenceid=101, filesize=11.6 K 2023-04-19 07:55:29,624 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for a8342afdd5fe71a59bc1e541995ded31 in 520ms, sequenceid=101, compaction requested=true 2023-04-19 07:55:29,624 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:29,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:29,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 07:55:29,624 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-19 07:55:29,624 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:29,625 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:29,626 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 98165 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:29,626 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/A is initiating minor compaction (all files) 2023-04-19 07:55:29,626 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:29,626 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/A in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:29,626 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/C is initiating minor compaction (all files) 2023-04-19 07:55:29,626 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/c4401770e22a4a0bbea48ebb6c7cd76a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0cab1c7be0294f568e6326530570a641, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/1514a2d9cf554ad68991c8ace074310f] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=95.9 K 2023-04-19 07:55:29,626 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/C in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:29,627 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/baf6e8e767804ef4b1de2cada7444128, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/79d6d41e177b4386a1b307f29ce94804, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/652aec2a0e1a4b3ba9e0d3eb3e8f778b] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=34.9 K 2023-04-19 07:55:29,627 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting c4401770e22a4a0bbea48ebb6c7cd76a, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681890926679 2023-04-19 07:55:29,627 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting baf6e8e767804ef4b1de2cada7444128, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681890926679 2023-04-19 07:55:29,627 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 0cab1c7be0294f568e6326530570a641, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681890926811 2023-04-19 07:55:29,627 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 79d6d41e177b4386a1b307f29ce94804, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681890926811 2023-04-19 07:55:29,628 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 1514a2d9cf554ad68991c8ace074310f, keycount=300, bloomtype=ROW, size=45.5 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1681890927956 2023-04-19 07:55:29,629 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 652aec2a0e1a4b3ba9e0d3eb3e8f778b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1681890929102 2023-04-19 07:55:29,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:29,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:29,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:29,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:29,634 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:29,634 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:29,634 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:29,634 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:29,635 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,635 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,637 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,637 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:29,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,638 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,638 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:29,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:29,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:29,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:29,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:29,640 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:29,640 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:29,640 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:29,642 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#C#compaction#316 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:29,644 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:29,644 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:29,645 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 07:55:29,645 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:29,645 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:29,645 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:29,645 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:29,645 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:29,645 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:29,653 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#A#compaction#315 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:29,664 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/7045629c307e4a0aac04e7a4176b02af as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7045629c307e4a0aac04e7a4176b02af 2023-04-19 07:55:29,673 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/C of a8342afdd5fe71a59bc1e541995ded31 into 7045629c307e4a0aac04e7a4176b02af(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:29,673 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:29,673 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/C, priority=13, startTime=1681890929624; duration=0sec 2023-04-19 07:55:29,673 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:29,673 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:29,675 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:29,675 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/B is initiating minor compaction (all files) 2023-04-19 07:55:29,675 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/B in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:29,676 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2acafd61e3c84f3a9a7879409d57fc4b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/141b751147c646339dbac5f178132572, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/fac58c4fa0e847978c94cb15134b7381] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=34.9 K 2023-04-19 07:55:29,676 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 2acafd61e3c84f3a9a7879409d57fc4b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1681890926679 2023-04-19 07:55:29,676 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 141b751147c646339dbac5f178132572, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681890926811 2023-04-19 07:55:29,677 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting fac58c4fa0e847978c94cb15134b7381, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1681890929102 2023-04-19 07:55:29,683 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#B#compaction#318 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:29,698 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/ac08b91f7a4d481f9afef1a5a18830a3 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/ac08b91f7a4d481f9afef1a5a18830a3 2023-04-19 07:55:29,703 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/B of a8342afdd5fe71a59bc1e541995ded31 into ac08b91f7a4d481f9afef1a5a18830a3(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:29,703 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:29,703 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/B, priority=13, startTime=1681890929624; duration=0sec 2023-04-19 07:55:29,703 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:29,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:29,751 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:29,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890989769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890989772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890989773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890989775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890989876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890989876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890989882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:29,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:29,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890989882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:30,082 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304195b39f4e25b034f3eb16b58b4c20b25e5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195b39f4e25b034f3eb16b58b4c20b25e5 2023-04-19 07:55:30,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:30,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890990083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:30,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:30,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890990083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:30,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:30,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890990089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:30,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:30,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890990089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:30,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:30,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890990389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:30,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:30,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890990389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:30,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:30,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890990394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:30,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:30,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890990394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:30,488 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/414135528ad44e0d9534bd5391496eab 2023-04-19 07:55:30,525 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/653557f8394f479198a58b2435fc1a3b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/653557f8394f479198a58b2435fc1a3b 2023-04-19 07:55:30,529 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/A of a8342afdd5fe71a59bc1e541995ded31 into 653557f8394f479198a58b2435fc1a3b(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:30,529 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:30,529 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/A, priority=13, startTime=1681890929624; duration=0sec 2023-04-19 07:55:30,529 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:30,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:30,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890990900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:30,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:30,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890990901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:30,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:30,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890990901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:30,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:30,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890990903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:30,924 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/4151d857cc36448cb83b220a8a4eefcf 2023-04-19 07:55:31,338 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/025400c0f2024e3f9830a61a3620e99f 2023-04-19 07:55:31,342 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/414135528ad44e0d9534bd5391496eab as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/414135528ad44e0d9534bd5391496eab 2023-04-19 07:55:31,346 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/414135528ad44e0d9534bd5391496eab, entries=150, sequenceid=117, filesize=25.1 K 2023-04-19 07:55:31,347 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/4151d857cc36448cb83b220a8a4eefcf as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/4151d857cc36448cb83b220a8a4eefcf 2023-04-19 07:55:31,350 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/4151d857cc36448cb83b220a8a4eefcf, entries=150, sequenceid=117, filesize=11.6 K 2023-04-19 07:55:31,351 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/025400c0f2024e3f9830a61a3620e99f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/025400c0f2024e3f9830a61a3620e99f 2023-04-19 07:55:31,355 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/025400c0f2024e3f9830a61a3620e99f, entries=150, sequenceid=117, filesize=11.6 K 2023-04-19 07:55:31,355 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for a8342afdd5fe71a59bc1e541995ded31 in 1710ms, sequenceid=117, compaction requested=false 2023-04-19 07:55:31,356 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:31,356 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:31,358 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,358 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:31,359 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,359 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:31,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:31,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e300bd[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:31,360 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:31,360 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,361 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,361 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:31,361 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:31,361 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:31,362 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,362 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:31,363 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,363 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:31,363 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:31,363 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:31,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:31,366 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:31,366 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:31,366 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:31,366 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:31,366 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:31,367 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,367 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,367 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:31,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:31,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:31,368 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,369 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:31,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,372 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,373 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,373 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,373 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:31,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:31,373 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:31,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:31,373 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:31,373 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:31,373 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,373 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:31,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:31,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:31,373 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:31,373 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,373 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:31,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:31,473 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:31,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:31,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:31,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:31,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:31,482 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:31,482 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:31,482 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:31,482 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:31,483 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,483 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,487 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,487 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,487 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:31,488 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:31,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:31,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:31,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:31,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:31,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:31,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:31,490 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:31,491 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:31,491 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:31,491 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:31,491 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:31,491 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:31,491 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:31,491 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 07:55:31,491 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:31,491 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:31,491 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:31,492 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:31,492 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:31,492 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:31,507 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419881f69e63b614ca3b1a49e46fe1f6803 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419881f69e63b614ca3b1a49e46fe1f6803 2023-04-19 07:55:31,514 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=136, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/0750e6d7cba64363a827cbc25b0e30f7 2023-04-19 07:55:31,530 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/2f4c719e937f49dabbc6412642d903f5 2023-04-19 07:55:31,546 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/79ec77abf22b42db93184969492c59af 2023-04-19 07:55:31,550 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/0750e6d7cba64363a827cbc25b0e30f7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0750e6d7cba64363a827cbc25b0e30f7 2023-04-19 07:55:31,569 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0750e6d7cba64363a827cbc25b0e30f7, entries=150, sequenceid=136, filesize=25.3 K 2023-04-19 07:55:31,572 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/2f4c719e937f49dabbc6412642d903f5 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2f4c719e937f49dabbc6412642d903f5 2023-04-19 07:55:31,578 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2f4c719e937f49dabbc6412642d903f5, entries=150, sequenceid=136, filesize=11.8 K 2023-04-19 07:55:31,579 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/79ec77abf22b42db93184969492c59af as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/79ec77abf22b42db93184969492c59af 2023-04-19 07:55:31,589 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/79ec77abf22b42db93184969492c59af, entries=150, sequenceid=136, filesize=11.8 K 2023-04-19 07:55:31,590 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for a8342afdd5fe71a59bc1e541995ded31 in 99ms, sequenceid=136, compaction requested=true 2023-04-19 07:55:31,590 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:31,590 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,591 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:31,591 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:31,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:31,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:31,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:31,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48d86417[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:31,594 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:31,594 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,595 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:31,596 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:31,596 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,596 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:31,597 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,597 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:31,597 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:31,597 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:31,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:31,600 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:31,600 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:31,600 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:31,600 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:31,600 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:31,603 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,603 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:31,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,603 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:31,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:31,604 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,605 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,607 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,607 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,607 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,607 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,607 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,607 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:31,607 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:31,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:31,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:31,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:31,607 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:31,607 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,608 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:31,608 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:31,608 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:31,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:31,708 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:31,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:31,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:31,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:31,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:31,714 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:31,714 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:31,714 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:31,714 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:31,714 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,714 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:31,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:31,718 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,718 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,718 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:31,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:31,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:31,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:31,719 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:31,719 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:31,719 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:31,720 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:31,720 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:31,720 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:31,720 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:31,721 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:31,722 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,722 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:31,722 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,722 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:31,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:31,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f52159e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:31,724 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:31,724 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,726 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:31,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:31,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:31,726 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,727 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:31,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:31,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:31,728 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:31,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:31,731 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:31,731 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:31,731 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:31,731 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:31,731 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:31,733 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:31,733 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,733 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:31,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:31,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:31,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:31,734 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,743 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:31,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:31,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:31,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:31,746 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:31,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,746 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:31,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:31,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:31,746 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:31,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:31,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,847 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:31,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:31,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:31,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:31,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:31,852 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:31,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:31,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:31,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:31,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,853 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,853 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:31,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:31,854 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,854 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:31,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:31,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:31,855 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:31,855 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:31,855 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:31,855 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:31,855 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:31,855 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:31,855 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:31,856 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:31,856 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:31,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:31,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:31,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:31,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:31,857 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,857 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:31,858 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,858 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:31,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:31,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cbeba82[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:31,860 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:31,860 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,862 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,862 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:31,862 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:31,862 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:31,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,862 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:31,863 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,863 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:31,863 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:31,863 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:31,866 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:31,866 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:31,866 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:31,866 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:31,866 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:31,868 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,868 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:31,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,868 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:31,868 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:31,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,870 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:31,872 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:31,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:31,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:31,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:31,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:31,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,873 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,873 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,873 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:31,873 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:31,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:31,873 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:31,973 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:31,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:31,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:31,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:31,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:31,988 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:31,988 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:31,988 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:31,988 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:31,989 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,989 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,990 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:31,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:31,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:31,990 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,991 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,991 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:31,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:31,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:31,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:31,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:31,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:31,992 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:55:31,992 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:31,992 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:31,992 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:31,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:31,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:31,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:31,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:31,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:31,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:31,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:31,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:31,996 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:31,996 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:31,996 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:31,996 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:31,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:31,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:31,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:31,996 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:31,999 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:31,999 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,000 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,000 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:32,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42c6af40[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:32,006 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:32,006 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,008 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,008 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:32,008 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:32,008 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:32,008 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,009 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:32,010 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,010 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:32,010 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,010 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:32,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:32,013 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:32,013 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:32,013 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:32,013 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:32,013 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:32,015 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,015 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,015 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:32,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,015 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:32,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:32,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,017 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,020 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:32,020 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:32,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:32,021 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,021 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,021 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,021 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,021 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,021 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,021 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,022 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:32,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:32,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:32,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:32,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:32,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:32,028 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:32,028 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:32,033 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419a0fd7bd9b737474faceee7a0a85044a2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419a0fd7bd9b737474faceee7a0a85044a2 2023-04-19 07:55:32,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681890932047, max timestamp=1681890932062Num uniques 859; 2023-04-19 07:55:32,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 07:55:32,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681890932047, max timestamp=1681890932062Num uniques 859; 2023-04-19 07:55:32,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 07:55:32,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681890932047, max timestamp=1681890932062Num uniques 859; 2023-04-19 07:55:32,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 07:55:32,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890992095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890992097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890992101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890992102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,128 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:32,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:32,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:32,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:32,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:32,132 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:32,132 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:32,132 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:32,132 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:32,132 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,132 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,133 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:32,133 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:32,134 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,134 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:32,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:32,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:32,135 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:32,135 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:32,135 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:32,135 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:32,135 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:32,135 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,135 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:32,135 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:32,135 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,135 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:32,135 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:32,135 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,135 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:32,135 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:32,140 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,140 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,141 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,141 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:32,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:32,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@493583d3[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:32,143 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:32,143 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,144 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,144 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,144 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:32,145 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:32,145 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:32,145 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:32,146 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:32,146 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:32,146 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:32,146 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:32,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:32,150 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:32,150 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:32,150 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:32,150 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:32,150 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:32,153 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,153 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:32,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:32,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:32,154 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,155 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,156 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,162 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:32,162 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:32,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:32,162 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:32,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:32,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:32,164 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,164 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,164 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,164 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,164 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,164 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,164 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,164 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:32,164 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:32,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:32,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:32,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890992204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890992204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890992207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890992212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,264 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:32,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:32,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:32,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:32,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:32,268 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:32,268 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:32,268 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:32,268 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:32,268 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,268 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,269 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:32,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:32,270 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,270 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,270 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:32,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:32,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:32,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:32,271 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,271 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:32,271 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:32,271 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:32,272 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,272 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:32,272 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:32,273 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,273 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,273 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,273 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:32,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:32,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1694918e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:32,275 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:32,275 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,276 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:32,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:32,276 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:32,276 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,276 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:32,278 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,278 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:32,278 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,278 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:32,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:32,282 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:32,282 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:32,282 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:32,282 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:32,282 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:32,283 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,283 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,283 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:32,284 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,285 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:32,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:32,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:32,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:32,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:32,287 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:32,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:32,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:32,288 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:32,288 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:32,388 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:32,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:32,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:32,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:32,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:32,392 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:32,392 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:32,392 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:32,392 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:32,392 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,392 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,393 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,393 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:32,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:32,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:32,394 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,394 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:32,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:32,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:32,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:32,396 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,396 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:32,396 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:32,396 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:32,396 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:32,399 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,399 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,399 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:32,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fd75374[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:32,401 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:32,401 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,402 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,403 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:32,403 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:32,403 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:32,404 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,404 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:32,405 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:32,405 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,405 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,405 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:32,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:32,407 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:32,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:32,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:32,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:32,407 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:32,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:32,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:32,410 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,411 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890992412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890992412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890992412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,418 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:32,418 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:32,418 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:32,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:32,418 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:32,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:32,419 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,419 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:32,419 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,419 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,419 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,419 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,419 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:32,419 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:32,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:32,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:32,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890992420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,447 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=149, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/1d85990923264f53b5142b928769c610 2023-04-19 07:55:32,519 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:32,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:32,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:32,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:32,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:32,523 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:32,523 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:32,523 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:32,523 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:32,523 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,523 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,525 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:32,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:32,526 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,526 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,526 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:32,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:32,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:32,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:32,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:32,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:32,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:32,532 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:32,570 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,570 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:32,570 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:32,570 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:32,570 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:32,574 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,574 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,574 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:32,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@481c4761[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:32,576 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:32,577 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,578 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:32,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:32,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:32,578 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,578 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:32,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:32,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:32,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:32,584 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:32,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:32,589 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:32,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:32,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:32,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:32,589 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:32,590 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,590 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,590 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:32,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:32,591 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,592 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,595 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,595 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:32,595 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:32,595 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,595 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,595 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,595 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,595 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,595 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:32,595 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:32,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:32,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:32,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:32,595 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:32,595 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:32,696 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:32,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:32,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:32,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:32,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:32,699 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:32,699 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:32,699 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:32,699 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:32,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,701 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:32,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:32,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:32,701 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,703 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,703 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:32,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:32,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:32,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:32,704 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:32,704 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:32,704 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:32,704 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:32,704 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:32,704 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,704 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:32,704 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:32,704 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,704 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:32,704 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:32,704 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,704 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:32,704 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:32,707 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,707 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,707 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:32,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dd27eaa[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:32,709 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:32,709 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,710 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,711 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:32,711 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:32,711 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:32,711 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,711 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:32,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:32,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:32,715 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:32,715 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:32,715 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:32,715 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:32,715 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:32,717 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,717 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:32,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,717 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,718 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,720 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,724 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:32,724 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:32,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:32,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:32,725 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,725 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,725 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,725 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,725 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,725 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,725 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,725 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:32,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:32,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:32,726 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:32,726 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:32,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890992720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890992720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890992720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890992733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:32,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47030 deadline: 1681890992749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,757 DEBUG [Thread-2221] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8225 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:55:32,826 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:32,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:32,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:32,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:32,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:32,829 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:32,829 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:32,829 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:32,829 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:32,830 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,830 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,831 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:32,831 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:32,831 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,832 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:32,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:32,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:32,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:32,833 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:32,833 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:32,833 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:32,835 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:32,835 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:32,835 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,835 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:32,835 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:32,835 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,835 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:32,835 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:32,835 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,835 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:32,835 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:32,836 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,836 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:32,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f78927f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:32,838 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:32,838 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,839 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:32,840 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,840 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:32,842 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:32,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:32,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:32,845 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:32,845 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:32,845 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:32,845 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:32,845 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:32,846 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,846 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:32,846 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:32,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:32,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,847 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,848 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,851 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,851 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:32,851 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:32,851 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:32,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:32,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:32,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:32,852 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,852 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,852 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,852 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:32,852 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,852 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:32,852 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:32,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:32,870 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/d6a1e8d9e60d413daa34a3209df52808 2023-04-19 07:55:32,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/d32fc42fd5184b698207d8ec181a3006 2023-04-19 07:55:32,886 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/1d85990923264f53b5142b928769c610 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/1d85990923264f53b5142b928769c610 2023-04-19 07:55:32,890 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/1d85990923264f53b5142b928769c610, entries=400, sequenceid=149, filesize=59.4 K 2023-04-19 07:55:32,891 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/d6a1e8d9e60d413daa34a3209df52808 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/d6a1e8d9e60d413daa34a3209df52808 2023-04-19 07:55:32,895 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/d6a1e8d9e60d413daa34a3209df52808, entries=150, sequenceid=149, filesize=11.8 K 2023-04-19 07:55:32,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/d32fc42fd5184b698207d8ec181a3006 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d32fc42fd5184b698207d8ec181a3006 2023-04-19 07:55:32,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d32fc42fd5184b698207d8ec181a3006, entries=150, sequenceid=149, filesize=11.8 K 2023-04-19 07:55:32,900 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for a8342afdd5fe71a59bc1e541995ded31 in 907ms, sequenceid=149, compaction requested=true 2023-04-19 07:55:32,900 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:32,900 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:32,900 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:55:32,900 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:55:32,900 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:32,900 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:32,901 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 138448 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:55:32,901 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48050 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:55:32,901 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/A is initiating minor compaction (all files) 2023-04-19 07:55:32,901 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/B is initiating minor compaction (all files) 2023-04-19 07:55:32,901 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/B in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,901 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/A in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,902 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/ac08b91f7a4d481f9afef1a5a18830a3, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/4151d857cc36448cb83b220a8a4eefcf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2f4c719e937f49dabbc6412642d903f5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/d6a1e8d9e60d413daa34a3209df52808] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=46.9 K 2023-04-19 07:55:32,902 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/653557f8394f479198a58b2435fc1a3b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/414135528ad44e0d9534bd5391496eab, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0750e6d7cba64363a827cbc25b0e30f7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/1d85990923264f53b5142b928769c610] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=135.2 K 2023-04-19 07:55:32,902 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting ac08b91f7a4d481f9afef1a5a18830a3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1681890929102 2023-04-19 07:55:32,902 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 653557f8394f479198a58b2435fc1a3b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1681890929102 2023-04-19 07:55:32,902 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 4151d857cc36448cb83b220a8a4eefcf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681890929123 2023-04-19 07:55:32,902 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 414135528ad44e0d9534bd5391496eab, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681890929123 2023-04-19 07:55:32,903 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 2f4c719e937f49dabbc6412642d903f5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681890929767 2023-04-19 07:55:32,903 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 0750e6d7cba64363a827cbc25b0e30f7, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681890929767 2023-04-19 07:55:32,903 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting d6a1e8d9e60d413daa34a3209df52808, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681890931974 2023-04-19 07:55:32,903 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 1d85990923264f53b5142b928769c610, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681890931972 2023-04-19 07:55:32,910 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#B#compaction#327 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:32,916 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#A#compaction#328 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:32,952 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:32,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:32,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:32,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:32,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:32,955 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:32,956 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:32,956 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:32,956 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:32,956 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,956 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:32,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:32,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:32,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:32,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:32,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:32,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:32,960 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:32,960 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:32,960 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-19 07:55:32,960 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:32,960 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:32,961 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:32,961 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:32,961 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:32,961 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:33,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:33,236 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:33,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890993245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890993246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890993247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890993254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,318 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/f37a7e34b38b440dafa9871fb5783ec7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f37a7e34b38b440dafa9871fb5783ec7 2023-04-19 07:55:33,322 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/B of a8342afdd5fe71a59bc1e541995ded31 into f37a7e34b38b440dafa9871fb5783ec7(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:33,322 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:33,322 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/B, priority=12, startTime=1681890932900; duration=0sec 2023-04-19 07:55:33,322 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:33,322 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:55:33,325 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48050 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:55:33,325 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/C is initiating minor compaction (all files) 2023-04-19 07:55:33,325 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/C in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:33,325 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7045629c307e4a0aac04e7a4176b02af, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/025400c0f2024e3f9830a61a3620e99f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/79ec77abf22b42db93184969492c59af, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d32fc42fd5184b698207d8ec181a3006] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=46.9 K 2023-04-19 07:55:33,326 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7045629c307e4a0aac04e7a4176b02af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1681890929102 2023-04-19 07:55:33,326 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 025400c0f2024e3f9830a61a3620e99f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681890929123 2023-04-19 07:55:33,326 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 79ec77abf22b42db93184969492c59af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681890929767 2023-04-19 07:55:33,327 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting d32fc42fd5184b698207d8ec181a3006, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681890931974 2023-04-19 07:55:33,334 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#C#compaction#330 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:33,340 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/84b128705ac441d19bc2ea876d5b4ad9 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/84b128705ac441d19bc2ea876d5b4ad9 2023-04-19 07:55:33,343 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/168b2ad4ed6b44919d2f3daf1f80ffc7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/168b2ad4ed6b44919d2f3daf1f80ffc7 2023-04-19 07:55:33,345 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/A of a8342afdd5fe71a59bc1e541995ded31 into 84b128705ac441d19bc2ea876d5b4ad9(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:33,345 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:33,345 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/A, priority=12, startTime=1681890932900; duration=0sec 2023-04-19 07:55:33,345 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:33,348 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/C of a8342afdd5fe71a59bc1e541995ded31 into 168b2ad4ed6b44919d2f3daf1f80ffc7(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:33,348 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:33,348 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/C, priority=12, startTime=1681890932900; duration=0sec 2023-04-19 07:55:33,348 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:33,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890993355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890993355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890993357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890993364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,383 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419703555bd367640388dea3faa451a2a9d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419703555bd367640388dea3faa451a2a9d 2023-04-19 07:55:33,387 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=183, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/7fe6824f8f694e6397c6f6c0efab578d 2023-04-19 07:55:33,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890993559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890993559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890993559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890993568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,797 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/5ffb657c120e4c8b9f8b62362cbcd738 2023-04-19 07:55:33,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890993862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890993862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890993862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:33,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:33,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890993875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,215 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/801e2626826d41c3a6938979614385ea 2023-04-19 07:55:34,219 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/7fe6824f8f694e6397c6f6c0efab578d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/7fe6824f8f694e6397c6f6c0efab578d 2023-04-19 07:55:34,222 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/7fe6824f8f694e6397c6f6c0efab578d, entries=150, sequenceid=183, filesize=25.3 K 2023-04-19 07:55:34,223 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/5ffb657c120e4c8b9f8b62362cbcd738 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/5ffb657c120e4c8b9f8b62362cbcd738 2023-04-19 07:55:34,227 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/5ffb657c120e4c8b9f8b62362cbcd738, entries=150, sequenceid=183, filesize=11.8 K 2023-04-19 07:55:34,228 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/801e2626826d41c3a6938979614385ea as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/801e2626826d41c3a6938979614385ea 2023-04-19 07:55:34,232 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/801e2626826d41c3a6938979614385ea, entries=150, sequenceid=183, filesize=11.8 K 2023-04-19 07:55:34,233 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for a8342afdd5fe71a59bc1e541995ded31 in 1273ms, sequenceid=183, compaction requested=false 2023-04-19 07:55:34,233 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:34,233 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:34,234 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:34,234 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:34,234 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,234 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:34,234 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:34,236 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:34,236 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:34,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:34,237 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:34,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:34,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ab12e17[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:34,239 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:34,239 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:34,240 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:34,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:34,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:34,240 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:34,240 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:34,240 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:34,240 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,240 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:34,241 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:34,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:34,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:34,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:34,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:34,245 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:34,245 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:34,245 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:34,245 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:34,245 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:34,250 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:34,250 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:34,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:34,250 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:34,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:34,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:34,251 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:34,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:34,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:34,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:34,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:34,254 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:34,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:34,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,260 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:34,261 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:34,261 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:34,261 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:34,261 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,261 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:34,261 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:34,261 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:34,261 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:34,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:34,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:34,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:34,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:34,261 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:34,261 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:34,261 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:34,362 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:34,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:34,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:34,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:34,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:34,365 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:34,365 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:34,365 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:34,365 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:34,365 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:34,365 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:34,366 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:34,366 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:34,367 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,367 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:34,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:34,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:34,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:34,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:34,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:34,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:34,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:34,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:34,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:34,368 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:34,368 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:34,368 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 07:55:34,369 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:34,369 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:34,369 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:34,369 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:34,369 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:34,369 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:34,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:34,371 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:34,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:34,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890994399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890934392, max timestamp=1681890934404Num uniques 859; 2023-04-19 07:55:34,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:55:34,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890934392, max timestamp=1681890934404Num uniques 859; 2023-04-19 07:55:34,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:55:34,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890934392, max timestamp=1681890934404Num uniques 859; 2023-04-19 07:55:34,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:55:34,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:34,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890994439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:34,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890994440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:34,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890994444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:34,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890994507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:34,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890994548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:34,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890994548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:34,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890994551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:34,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890994714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:34,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890994754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:34,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890994755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:34,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890994758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:34,783 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304199f693f59288a450995a5d0fa997fd68d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199f693f59288a450995a5d0fa997fd68d 2023-04-19 07:55:35,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:35,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890995021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:35,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890995061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:35,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890995061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:35,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890995063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,189 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=197, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/595a70d3cc0e48019c7fdf5af7882aba 2023-04-19 07:55:35,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:35,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890995534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:35,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890995573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:35,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890995574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:35,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890995577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,599 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/2ef6d9baabc34bd3a903dc52e1e91780 2023-04-19 07:55:35,614 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/7eb38b2b70974df8ab2f4bc0bfc01eea 2023-04-19 07:55:35,618 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/595a70d3cc0e48019c7fdf5af7882aba as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/595a70d3cc0e48019c7fdf5af7882aba 2023-04-19 07:55:35,621 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/595a70d3cc0e48019c7fdf5af7882aba, entries=150, sequenceid=197, filesize=25.3 K 2023-04-19 07:55:35,622 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/2ef6d9baabc34bd3a903dc52e1e91780 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2ef6d9baabc34bd3a903dc52e1e91780 2023-04-19 07:55:35,626 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2ef6d9baabc34bd3a903dc52e1e91780, entries=150, sequenceid=197, filesize=11.8 K 2023-04-19 07:55:35,626 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-19 07:55:35,628 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/7eb38b2b70974df8ab2f4bc0bfc01eea as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7eb38b2b70974df8ab2f4bc0bfc01eea 2023-04-19 07:55:35,636 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7eb38b2b70974df8ab2f4bc0bfc01eea, entries=150, sequenceid=197, filesize=11.8 K 2023-04-19 07:55:35,637 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a8342afdd5fe71a59bc1e541995ded31 in 1269ms, sequenceid=197, compaction requested=true 2023-04-19 07:55:35,637 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:35,637 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:35,637 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:35,638 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:35,638 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,638 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:35,638 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:35,639 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:35,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:35,639 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:35,640 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:35,640 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:35,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:35,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:35,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@303c556f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:35,645 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:35,645 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:35,646 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:35,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:35,646 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:35,646 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:35,646 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:35,646 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,647 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:35,648 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:35,648 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:35,649 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:35,649 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:35,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:35,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:35,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:35,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:35,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:35,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:35,654 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:35,654 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:35,654 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:35,654 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:35,654 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:35,656 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:35,656 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:35,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:35,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:35,656 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:35,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:35,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:35,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:35,660 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,664 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,666 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:35,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:35,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:35,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:35,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:35,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:35,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:35,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:35,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:35,667 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:35,667 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:35,667 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:35,666 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:35,667 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:35,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:35,767 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:35,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:35,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:35,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:35,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:35,779 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:35,779 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:35,779 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:35,779 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:35,780 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:35,780 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:35,782 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,782 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:35,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:35,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:35,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:35,782 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:35,783 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:35,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:35,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:35,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:35,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:35,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:35,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:35,784 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:35,784 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:35,784 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:35,785 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:35,785 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:35,785 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:35,785 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:35,785 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-19 07:55:35,786 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:35,786 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:35,786 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:35,786 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:35,786 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:35,786 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:35,810 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419fe3ee81ed4664e5a85e33795a6bac1ff to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419fe3ee81ed4664e5a85e33795a6bac1ff 2023-04-19 07:55:36,221 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=232, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/20e8e740b83c493e922670bca8ec9947 2023-04-19 07:55:36,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:36,555 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:36,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:36,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890996583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:36,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46972 deadline: 1681890996585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:36,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890996588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:36,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46946 deadline: 1681890996593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,632 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/19695ef5dba94efbaff425d849886ff5 2023-04-19 07:55:36,642 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/d183a255299a4c8688579f77faf59d48 2023-04-19 07:55:36,645 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/20e8e740b83c493e922670bca8ec9947 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/20e8e740b83c493e922670bca8ec9947 2023-04-19 07:55:36,648 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/20e8e740b83c493e922670bca8ec9947, entries=150, sequenceid=232, filesize=25.3 K 2023-04-19 07:55:36,649 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/19695ef5dba94efbaff425d849886ff5 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/19695ef5dba94efbaff425d849886ff5 2023-04-19 07:55:36,652 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/19695ef5dba94efbaff425d849886ff5, entries=150, sequenceid=232, filesize=11.8 K 2023-04-19 07:55:36,653 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/d183a255299a4c8688579f77faf59d48 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d183a255299a4c8688579f77faf59d48 2023-04-19 07:55:36,656 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d183a255299a4c8688579f77faf59d48, entries=150, sequenceid=232, filesize=11.8 K 2023-04-19 07:55:36,657 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for a8342afdd5fe71a59bc1e541995ded31 in 872ms, sequenceid=232, compaction requested=true 2023-04-19 07:55:36,657 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:36,657 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:36,657 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:36,657 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:36,657 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,657 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:36,657 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:36,659 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,659 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:36,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:36,659 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,659 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:36,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:36,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:36,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:36,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:36,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e255aec[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:36,661 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:36,661 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,662 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,662 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:36,662 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:36,663 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:36,663 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,663 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:36,664 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:36,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:36,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:36,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:36,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:36,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:36,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:36,667 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:36,667 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:36,667 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:36,667 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:36,667 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:36,669 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,669 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:36,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,669 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:36,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:36,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:36,670 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:36,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,671 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:36,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:36,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:36,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,673 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:36,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:36,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:36,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:36,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:36,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:36,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:36,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:36,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,674 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:36,674 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:36,674 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:36,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:36,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:36,693 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 07:55:36,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:36,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:36,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:36,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:36,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:36,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:36,708 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041916fd41c0453949afabe874dcee56b957 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041916fd41c0453949afabe874dcee56b957 2023-04-19 07:55:36,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:36,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46996 deadline: 1681890996734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:36,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681890996741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:36,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890996742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,774 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:36,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:36,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:36,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:36,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:36,777 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:36,777 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:36,777 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:36,777 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:36,777 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,777 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:36,778 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,778 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:36,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:36,779 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,779 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:36,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:36,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:36,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:36,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:36,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:36,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:36,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:36,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:36,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:36,780 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:36,780 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:36,780 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:36,780 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:36,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:36,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:36,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:36,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:36,782 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,782 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:36,782 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,782 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:36,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:36,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:36,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5515da3d[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:36,783 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:36,783 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,785 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,785 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:36,785 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,786 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:36,785 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:36,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:36,787 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:36,787 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,787 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:36,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:36,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:36,787 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:36,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:36,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:36,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:36,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:36,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:36,789 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:36,789 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:36,789 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:36,789 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:36,789 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:36,790 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,790 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,790 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:36,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:36,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:36,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:36,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:36,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:36,791 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:36,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,794 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,800 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:36,800 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,800 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:36,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:36,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:36,800 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:36,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:36,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:36,800 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:36,800 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:36,800 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:36,800 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:36,800 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,800 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,800 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:36,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:36,800 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:36,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681890996843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:36,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681890996847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:36,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46994 deadline: 1681890996848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,900 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:36,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:36,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:36,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:36,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:36,904 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:36,904 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:36,904 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:36,904 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:36,905 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,905 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:36,906 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:36,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:36,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:36,906 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:36,907 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,907 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:36,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:36,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:36,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:36,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:36,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:36,908 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:36,908 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:36,908 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:36,908 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:36,908 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:36,908 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:36,908 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:36,908 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:36,908 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:36,909 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:36,910 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,910 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:36,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:36,910 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,910 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:36,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:36,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46c7971e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:36,912 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:36,912 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,914 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,914 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:36,914 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:36,914 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,914 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:36,914 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:36,916 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,916 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:36,916 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:36,916 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:36,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:36,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:36,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:36,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:36,920 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:36,920 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:36,920 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:36,920 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:36,921 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:36,922 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,922 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:36,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:36,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,922 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:36,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:36,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:36,923 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:36,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:36,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,924 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,926 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,926 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:36,926 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:36,926 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:36,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:36,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:36,926 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:36,926 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:36,926 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,926 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:36,926 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:36,926 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:36,927 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:36,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:36,926 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:36,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:36,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,027 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:37,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:37,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:37,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:37,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:37,039 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:37,039 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,039 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:37,039 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:37,039 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,040 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,047 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,048 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,048 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:37,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:37,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:37,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:37,052 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:37,052 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,052 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:37,055 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:37,055 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:37,055 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,055 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:37,055 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:37,055 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,055 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:37,055 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:37,055 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,055 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:37,055 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:37,058 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:37,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681890997049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:37,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14823584[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:37,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:37,060 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:37,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681890997051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,060 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:37,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681890997052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,061 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,061 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:37,061 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,061 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:37,061 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:37,061 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:37,062 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:37,062 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:37,062 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,062 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:37,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:37,065 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:37,065 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:37,065 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:37,065 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:37,065 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:37,068 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,068 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,068 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:37,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,068 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,069 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,071 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,071 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:37,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:37,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:37,071 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:37,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:37,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,072 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:37,073 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:37,113 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=244, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/4a8e3be4087b44ef83fd3c63b6becadd 2023-04-19 07:55:37,173 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:37,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:37,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:37,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:37,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:37,180 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:37,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:37,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:37,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,181 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,181 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,182 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,182 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:37,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:37,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:37,184 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:37,184 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,184 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:37,188 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:37,188 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:37,188 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,188 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:37,189 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:37,189 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:37,190 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,190 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,191 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,191 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:37,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3054ffc2[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:37,192 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:37,193 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,194 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,195 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:37,195 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,195 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:37,195 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:37,195 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:37,196 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:37,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:37,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:37,198 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:37,198 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:37,198 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:37,198 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:37,198 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:37,200 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,200 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,200 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:37,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,201 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,202 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,204 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,204 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,204 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,204 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:37,204 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,204 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:37,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:37,204 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,204 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:37,204 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,205 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:37,204 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,205 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:37,305 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:37,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:37,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:37,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:37,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:37,308 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:37,308 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,308 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:37,308 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:37,309 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,309 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,313 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,313 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,313 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,313 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:37,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:37,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:37,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:37,314 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:37,314 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,314 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:37,315 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,315 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:37,315 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:37,315 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:37,316 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,316 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,316 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,316 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:37,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4beb0fc7[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:37,318 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:37,318 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,320 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,320 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:37,320 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:37,320 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,320 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:37,320 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:37,322 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,322 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:37,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:37,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:37,325 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:37,325 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:37,325 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:37,325 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:37,325 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:37,326 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,327 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,327 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:37,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,328 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,329 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,331 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,331 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,331 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,331 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:37,331 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:37,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:37,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:37,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,331 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,331 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,331 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,331 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,331 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,331 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:37,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:37,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,332 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:37,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:37,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681890997361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:37,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681890997362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:37,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681890997363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,432 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:37,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:37,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:37,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:37,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:37,435 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:37,435 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,435 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:37,435 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:37,436 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,436 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,437 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,437 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,437 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,437 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:37,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:37,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:37,439 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,439 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:37,439 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:37,439 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:37,441 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,442 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,442 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:37,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@316e1ec3[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:37,443 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:37,443 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,444 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,444 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:37,444 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,445 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:37,444 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:37,445 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:37,446 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:37,446 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:37,446 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:37,446 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:37,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:37,448 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:37,448 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:37,448 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:37,449 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:37,449 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:37,450 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,450 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,450 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:37,451 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,452 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,455 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,455 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,455 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,455 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,455 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:37,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,455 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:37,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:37,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:37,455 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,456 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:37,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,456 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,456 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:37,456 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,456 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:37,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/1e3d09bb12264bd19feb90255b56f90a 2023-04-19 07:55:37,556 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:37,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:37,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:37,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:37,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:37,559 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:37,559 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,559 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:37,559 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:37,560 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,560 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,561 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,561 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,561 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,561 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:37,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:37,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:37,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:37,563 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,563 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:37,563 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:37,563 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,563 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:37,563 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:37,563 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,563 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:37,563 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:37,564 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,564 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,565 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,565 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3445337f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:37,566 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:37,566 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,567 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:37,567 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:37,567 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,568 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:37,567 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:37,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:37,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:37,575 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,575 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:37,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:37,578 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:37,578 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:37,578 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:37,578 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:37,578 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:37,579 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,579 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,579 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:37,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,580 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,581 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,583 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:37,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:37,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:37,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,584 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:37,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:37,584 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:37,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:37,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,684 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:37,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:37,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:37,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:37,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:37,687 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:37,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:37,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:37,688 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,688 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,689 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,689 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,689 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:37,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:37,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:37,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:37,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:37,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:37,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:37,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:37,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:37,692 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,692 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:37,692 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:37,692 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:37,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:37,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:37,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:37,697 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,697 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,698 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,698 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:37,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@237860da[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:37,699 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:37,699 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,701 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:37,701 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:37,701 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:37,702 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,702 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:37,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,703 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:37,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,703 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:37,703 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:37,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:37,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:37,705 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:37,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:37,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:37,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:37,706 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:37,708 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,708 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,708 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:37,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:37,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,709 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,710 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,712 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:37,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:37,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:37,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:37,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:37,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,713 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:37,713 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:37,813 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:37,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:37,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:37,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:37,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:37,816 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:37,816 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,816 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:37,817 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:37,817 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,817 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,818 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,818 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,818 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:37,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:37,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:37,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:37,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:37,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:37,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:37,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:37,820 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,820 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:37,820 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:37,820 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:37,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:37,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:37,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:37,821 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,821 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,822 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,822 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:37,823 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:37,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30fec645[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:37,823 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,825 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,825 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:37,825 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:37,825 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:37,825 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,826 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:37,826 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,827 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:37,827 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:37,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:37,829 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:37,829 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:37,829 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:37,829 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:37,829 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:37,830 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,830 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:37,830 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:37,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:37,831 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:37,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,832 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:37,834 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:37,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:37,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:37,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:37,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:37,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:37,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,835 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,835 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:37,835 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:37,835 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,835 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:37,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:37,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681890997866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:37,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681890997869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:37,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681890997870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,935 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:37,936 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/3da258ce40f74355a05279be076fb9fd 2023-04-19 07:55:37,939 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/4a8e3be4087b44ef83fd3c63b6becadd as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/4a8e3be4087b44ef83fd3c63b6becadd 2023-04-19 07:55:37,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:37,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/4a8e3be4087b44ef83fd3c63b6becadd, entries=200, sequenceid=244, filesize=32.1 K 2023-04-19 07:55:37,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/1e3d09bb12264bd19feb90255b56f90a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/1e3d09bb12264bd19feb90255b56f90a 2023-04-19 07:55:37,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:37,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:37,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:37,949 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:37,949 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,949 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:37,949 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:37,949 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,949 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,949 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/1e3d09bb12264bd19feb90255b56f90a, entries=150, sequenceid=244, filesize=11.8 K 2023-04-19 07:55:37,950 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/3da258ce40f74355a05279be076fb9fd as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3da258ce40f74355a05279be076fb9fd 2023-04-19 07:55:37,956 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:37,956 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:37,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,956 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:37,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:37,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:37,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:37,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:37,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:37,958 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:37,958 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:37,958 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:37,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3da258ce40f74355a05279be076fb9fd, entries=150, sequenceid=244, filesize=11.8 K 2023-04-19 07:55:37,962 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a8342afdd5fe71a59bc1e541995ded31 in 1269ms, sequenceid=244, compaction requested=true 2023-04-19 07:55:37,963 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:37,963 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:37,963 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:37,963 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:55:37,963 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:55:37,963 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:37,964 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:37,964 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:37,964 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,964 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:37,964 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 07:55:37,964 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60522 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:55:37,964 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/B is initiating minor compaction (all files) 2023-04-19 07:55:37,964 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:37,964 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/B in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,964 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:37,964 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f37a7e34b38b440dafa9871fb5783ec7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/5ffb657c120e4c8b9f8b62362cbcd738, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2ef6d9baabc34bd3a903dc52e1e91780, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/19695ef5dba94efbaff425d849886ff5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/1e3d09bb12264bd19feb90255b56f90a] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=59.1 K 2023-04-19 07:55:37,964 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:37,964 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136812 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:55:37,964 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:37,965 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/A is initiating minor compaction (all files) 2023-04-19 07:55:37,965 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:37,965 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/A in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:37,965 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:37,965 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/84b128705ac441d19bc2ea876d5b4ad9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/7fe6824f8f694e6397c6f6c0efab578d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/595a70d3cc0e48019c7fdf5af7882aba, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/20e8e740b83c493e922670bca8ec9947, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/4a8e3be4087b44ef83fd3c63b6becadd] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=133.6 K 2023-04-19 07:55:37,965 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting f37a7e34b38b440dafa9871fb5783ec7, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681890931974 2023-04-19 07:55:37,965 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 84b128705ac441d19bc2ea876d5b4ad9, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681890931974 2023-04-19 07:55:37,965 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 5ffb657c120e4c8b9f8b62362cbcd738, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1681890932095 2023-04-19 07:55:37,966 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 7fe6824f8f694e6397c6f6c0efab578d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1681890932095 2023-04-19 07:55:37,966 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 2ef6d9baabc34bd3a903dc52e1e91780, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681890933244 2023-04-19 07:55:37,966 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 595a70d3cc0e48019c7fdf5af7882aba, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681890933244 2023-04-19 07:55:37,966 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 19695ef5dba94efbaff425d849886ff5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681890934435 2023-04-19 07:55:37,967 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 20e8e740b83c493e922670bca8ec9947, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681890934435 2023-04-19 07:55:37,967 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 1e3d09bb12264bd19feb90255b56f90a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681890936569 2023-04-19 07:55:37,967 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 4a8e3be4087b44ef83fd3c63b6becadd, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681890936569 2023-04-19 07:55:37,978 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#B#compaction#344 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:37,983 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304191e92f1c6acac47a491d94d7ceef6f34a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191e92f1c6acac47a491d94d7ceef6f34a 2023-04-19 07:55:37,988 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#A#compaction#343 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:37,997 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/b32b11dcfd2c448187be39ec1c6f8230 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/b32b11dcfd2c448187be39ec1c6f8230 2023-04-19 07:55:38,001 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/B of a8342afdd5fe71a59bc1e541995ded31 into b32b11dcfd2c448187be39ec1c6f8230(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:38,001 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:38,001 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/B, priority=11, startTime=1681890937963; duration=0sec 2023-04-19 07:55:38,001 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:38,001 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:55:38,002 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60522 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:55:38,003 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/C is initiating minor compaction (all files) 2023-04-19 07:55:38,003 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/C in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:38,003 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/168b2ad4ed6b44919d2f3daf1f80ffc7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/801e2626826d41c3a6938979614385ea, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7eb38b2b70974df8ab2f4bc0bfc01eea, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d183a255299a4c8688579f77faf59d48, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3da258ce40f74355a05279be076fb9fd] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=59.1 K 2023-04-19 07:55:38,003 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 168b2ad4ed6b44919d2f3daf1f80ffc7, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681890931974 2023-04-19 07:55:38,003 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 801e2626826d41c3a6938979614385ea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1681890932095 2023-04-19 07:55:38,004 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7eb38b2b70974df8ab2f4bc0bfc01eea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681890933244 2023-04-19 07:55:38,004 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting d183a255299a4c8688579f77faf59d48, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681890934435 2023-04-19 07:55:38,004 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 3da258ce40f74355a05279be076fb9fd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681890936569 2023-04-19 07:55:38,013 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#C#compaction#345 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:38,396 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=264, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/ebef4ca79f124216b0afafaf57800f20 2023-04-19 07:55:38,410 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/fe563a19ed24470990018ba68ff7a98c 2023-04-19 07:55:38,415 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/3d3709008af8459389393d9aa0daedc1 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/3d3709008af8459389393d9aa0daedc1 2023-04-19 07:55:38,421 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/A of a8342afdd5fe71a59bc1e541995ded31 into 3d3709008af8459389393d9aa0daedc1(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:38,421 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:38,421 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/A, priority=11, startTime=1681890937963; duration=0sec 2023-04-19 07:55:38,421 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:38,422 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/d1e7db8ddd5843dbaabb71bdc0c6604e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d1e7db8ddd5843dbaabb71bdc0c6604e 2023-04-19 07:55:38,425 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/ff81b9d62b1a4b788f5e0ba65e73ded4 2023-04-19 07:55:38,429 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/ebef4ca79f124216b0afafaf57800f20 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/ebef4ca79f124216b0afafaf57800f20 2023-04-19 07:55:38,430 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/C of a8342afdd5fe71a59bc1e541995ded31 into d1e7db8ddd5843dbaabb71bdc0c6604e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:38,430 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:38,430 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/C, priority=11, startTime=1681890937963; duration=0sec 2023-04-19 07:55:38,430 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:38,434 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/ebef4ca79f124216b0afafaf57800f20, entries=150, sequenceid=264, filesize=25.4 K 2023-04-19 07:55:38,435 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/fe563a19ed24470990018ba68ff7a98c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/fe563a19ed24470990018ba68ff7a98c 2023-04-19 07:55:38,438 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/fe563a19ed24470990018ba68ff7a98c, entries=150, sequenceid=264, filesize=11.9 K 2023-04-19 07:55:38,439 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/ff81b9d62b1a4b788f5e0ba65e73ded4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/ff81b9d62b1a4b788f5e0ba65e73ded4 2023-04-19 07:55:38,442 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/ff81b9d62b1a4b788f5e0ba65e73ded4, entries=150, sequenceid=264, filesize=11.9 K 2023-04-19 07:55:38,442 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for a8342afdd5fe71a59bc1e541995ded31 in 478ms, sequenceid=264, compaction requested=false 2023-04-19 07:55:38,442 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:38,442 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:38,442 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:38,442 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:38,442 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,443 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:38,443 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:38,444 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,444 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:38,445 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,445 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:38,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:38,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71a75e9f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:38,446 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:38,446 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,447 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,447 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:38,447 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,447 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:38,447 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:38,448 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:38,449 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,449 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:38,449 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:38,449 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:38,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:38,451 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:38,452 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:38,452 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:38,452 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:38,452 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:38,454 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,454 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:38,455 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:38,455 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:38,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:38,456 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,458 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,458 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:38,458 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:38,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:38,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:38,458 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,458 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,459 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,459 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,458 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:38,458 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:38,459 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:38,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:38,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:38,459 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:38,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:38,459 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:38,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:38,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:38,559 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:38,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:38,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:38,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:38,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:38,563 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:38,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:38,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:38,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:38,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:38,564 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:38,564 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:38,565 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,565 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:38,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:38,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:38,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:38,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:38,566 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:38,566 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:38,566 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:38,566 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:38,566 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:38,566 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:38,566 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:38,566 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:38,567 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:38,567 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:38,567 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:38,567 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,567 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:38,567 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:38,568 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,568 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:38,568 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,568 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:38,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:38,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ce4551f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:38,570 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:38,570 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,573 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:38,573 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:38,573 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:38,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:38,577 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:38,577 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:38,577 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,577 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:38,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:38,579 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:38,579 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:38,579 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:38,580 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:38,580 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:38,581 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:38,581 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,581 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:38,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:38,581 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:38,583 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:38,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:38,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:38,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:38,585 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:38,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:38,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:38,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:38,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:38,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:38,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:38,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,586 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,586 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:38,586 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:38,586 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:38,586 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:38,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:38,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:38,686 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:38,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:38,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:38,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:38,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:38,689 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:38,689 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:38,690 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:38,690 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:38,690 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,690 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:38,691 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,691 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:38,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:38,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:38,691 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:38,692 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:38,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:38,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:38,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:38,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:38,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:38,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:38,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:38,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:38,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:38,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:38,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:38,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:38,693 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:38,694 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:38,694 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-19 07:55:38,694 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:38,694 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:38,694 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:38,694 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:38,694 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:38,694 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:38,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:38,898 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:38,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890938919, max timestamp=1681890938929Num uniques 859; 2023-04-19 07:55:38,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:55:38,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890938919, max timestamp=1681890938929Num uniques 859; 2023-04-19 07:55:38,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890938919, max timestamp=1681890938929Num uniques 859; 2023-04-19 07:55:38,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:55:38,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:55:38,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890938945, max timestamp=1681890938964Num uniques 1018; 2023-04-19 07:55:38,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 07:55:38,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890938945, max timestamp=1681890938964Num uniques 1018; 2023-04-19 07:55:38,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 07:55:38,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890938945, max timestamp=1681890938964Num uniques 1018; 2023-04-19 07:55:38,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 07:55:39,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890938989, max timestamp=1681890939006Num uniques 1127; 2023-04-19 07:55:39,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890938989, max timestamp=1681890939006Num uniques 1127; 2023-04-19 07:55:39,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890938989, max timestamp=1681890939006Num uniques 1127; 2023-04-19 07:55:39,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-19 07:55:39,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-19 07:55:39,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-19 07:55:39,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681890939023, max timestamp=1681890939035Num uniques 1086; 2023-04-19 07:55:39,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681890939023, max timestamp=1681890939035Num uniques 1086; 2023-04-19 07:55:39,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681890939023, max timestamp=1681890939035Num uniques 1086; 2023-04-19 07:55:39,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-19 07:55:39,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-19 07:55:39,038 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-19 07:55:39,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681890939053, max timestamp=1681890939063Num uniques 1095; 2023-04-19 07:55:39,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-19 07:55:39,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681890939053, max timestamp=1681890939063Num uniques 1095; 2023-04-19 07:55:39,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-19 07:55:39,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681890939053, max timestamp=1681890939063Num uniques 1095; 2023-04-19 07:55:39,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-19 07:55:39,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1681890939080, max timestamp=1681890939098Num uniques 1154; 2023-04-19 07:55:39,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52844, new segment data size=9334, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-19 07:55:39,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681890939080, max timestamp=1681890939098Num uniques 1154; 2023-04-19 07:55:39,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-19 07:55:39,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1681890939080, max timestamp=1681890939098Num uniques 1154; 2023-04-19 07:55:39,101 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52844, new segment data size=9334, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-19 07:55:39,108 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419cb279b6a91c94d68bb342c45140e956d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419cb279b6a91c94d68bb342c45140e956d 2023-04-19 07:55:39,113 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=273, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/46861bc650134480b411cf04ac7f1331 2023-04-19 07:55:39,123 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/1949da7159ca4e148fa72b2513158f44 2023-04-19 07:55:39,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681890939115, max timestamp=1681890939138Num uniques 1063; 2023-04-19 07:55:39,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681890939115, max timestamp=1681890939138Num uniques 1063; 2023-04-19 07:55:39,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-19 07:55:39,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-19 07:55:39,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681890939115, max timestamp=1681890939138Num uniques 1063; 2023-04-19 07:55:39,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-19 07:55:39,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10152, totalHeapSize=23328, min timestamp=1681890939164, max timestamp=1681890939177Num uniques 1122; 2023-04-19 07:55:39,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:39,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51372, new segment data size=10152, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-19 07:55:39,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681890999172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10157, totalHeapSize=23328, min timestamp=1681890939164, max timestamp=1681890939177Num uniques 1122; 2023-04-19 07:55:39,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51377, new segment data size=10157, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-19 07:55:39,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10157, totalHeapSize=23328, min timestamp=1681890939164, max timestamp=1681890939177Num uniques 1122; 2023-04-19 07:55:39,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51377, new segment data size=10157, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-19 07:55:39,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681890939203, max timestamp=1681890939222Num uniques 1081; 2023-04-19 07:55:39,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-19 07:55:39,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681890939203, max timestamp=1681890939222Num uniques 1081; 2023-04-19 07:55:39,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-19 07:55:39,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681890939203, max timestamp=1681890939222Num uniques 1081; 2023-04-19 07:55:39,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-19 07:55:39,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681890939250, max timestamp=1681890939262Num uniques 1090; 2023-04-19 07:55:39,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681890939250, max timestamp=1681890939262Num uniques 1090; 2023-04-19 07:55:39,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-19 07:55:39,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-19 07:55:39,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681890939250, max timestamp=1681890939262Num uniques 1090; 2023-04-19 07:55:39,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-19 07:55:39,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681890939296, max timestamp=1681890939305Num uniques 1099; 2023-04-19 07:55:39,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681890939296, max timestamp=1681890939305Num uniques 1099; 2023-04-19 07:55:39,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-19 07:55:39,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681890939296, max timestamp=1681890939305Num uniques 1099; 2023-04-19 07:55:39,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-19 07:55:39,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-19 07:55:39,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681890939334, max timestamp=1681890939369Num uniques 1108; 2023-04-19 07:55:39,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-19 07:55:39,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681890939334, max timestamp=1681890939369Num uniques 1108; 2023-04-19 07:55:39,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-19 07:55:39,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681890939334, max timestamp=1681890939369Num uniques 1108; 2023-04-19 07:55:39,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-19 07:55:39,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=317, cellSize=14512, totalHeapSize=33208, min timestamp=1681890939386, max timestamp=1681890939402Num uniques 1067; 2023-04-19 07:55:39,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=14512, suffix heap size=138936,new segment heap size=33208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=317 2023-04-19 07:55:39,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=317, cellSize=14512, totalHeapSize=33208, min timestamp=1681890939386, max timestamp=1681890939402Num uniques 1067; 2023-04-19 07:55:39,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=317, cellSize=14512, totalHeapSize=33208, min timestamp=1681890939386, max timestamp=1681890939402Num uniques 1067; 2023-04-19 07:55:39,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=14512, suffix heap size=138936,new segment heap size=33208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=317 2023-04-19 07:55:39,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=14512, suffix heap size=138936,new segment heap size=33208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=317 2023-04-19 07:55:39,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681890939439, max timestamp=1681890939446Num uniques 1176; 2023-04-19 07:55:39,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53856, new segment data size=10346, suffix heap size=150272,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1176, new segment cells count=226 2023-04-19 07:55:39,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681890939439, max timestamp=1681890939446Num uniques 1176; 2023-04-19 07:55:39,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53856, new segment data size=10346, suffix heap size=150272,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1176, new segment cells count=226 2023-04-19 07:55:39,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1681890939439, max timestamp=1681890939446Num uniques 1176; 2023-04-19 07:55:39,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53856, new segment data size=10346, suffix heap size=150272,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1176, new segment cells count=226 2023-04-19 07:55:39,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681890939469, max timestamp=1681890939480Num uniques 1085; 2023-04-19 07:55:39,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-19 07:55:39,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681890939469, max timestamp=1681890939480Num uniques 1085; 2023-04-19 07:55:39,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-19 07:55:39,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681890939469, max timestamp=1681890939480Num uniques 1085; 2023-04-19 07:55:39,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-19 07:55:39,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=344, cellSize=15754, totalHeapSize=36016, min timestamp=1681890939506, max timestamp=1681890939519Num uniques 1094; 2023-04-19 07:55:39,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=15754, suffix heap size=141744,new segment heap size=36016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=344 2023-04-19 07:55:39,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681890939506, max timestamp=1681890939519Num uniques 1094; 2023-04-19 07:55:39,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-04-19 07:55:39,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1681890939506, max timestamp=1681890939519Num uniques 1094; 2023-04-19 07:55:39,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-04-19 07:55:39,532 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/0b11afc3f6484c52a9f3afc5497676d0 2023-04-19 07:55:39,536 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/46861bc650134480b411cf04ac7f1331 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/46861bc650134480b411cf04ac7f1331 2023-04-19 07:55:39,540 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/46861bc650134480b411cf04ac7f1331, entries=100, sequenceid=273, filesize=18.6 K 2023-04-19 07:55:39,541 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/1949da7159ca4e148fa72b2513158f44 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/1949da7159ca4e148fa72b2513158f44 2023-04-19 07:55:39,544 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/1949da7159ca4e148fa72b2513158f44, entries=50, sequenceid=273, filesize=7.1 K 2023-04-19 07:55:39,545 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/0b11afc3f6484c52a9f3afc5497676d0 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/0b11afc3f6484c52a9f3afc5497676d0 2023-04-19 07:55:39,548 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/0b11afc3f6484c52a9f3afc5497676d0, entries=50, sequenceid=273, filesize=7.1 K 2023-04-19 07:55:39,548 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=143.13 KB/146560 for a8342afdd5fe71a59bc1e541995ded31 in 854ms, sequenceid=273, compaction requested=true 2023-04-19 07:55:39,548 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:39,548 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:39,548 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:39,548 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:39,549 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,549 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:39,549 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:39,550 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,550 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:39,550 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,550 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:39,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:39,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:39,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c7a96e9[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:39,552 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:39,552 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,553 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,553 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:39,553 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:39,553 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,553 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:39,553 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:39,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:39,555 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=149.83 KB heapSize=417.25 KB 2023-04-19 07:55:39,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:39,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:39,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:39,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:39,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:39,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:39,555 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:39,555 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,555 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:39,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:39,555 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:39,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:39,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:39,558 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:39,558 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:39,558 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:39,558 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:39,558 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:39,561 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,561 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:39,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,562 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:39,562 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:39,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:39,566 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681890999558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,566 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,566 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,566 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:39,566 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:39,566 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:39,566 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:39,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:39,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:39,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:39,567 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:39,567 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:39,566 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:39,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:39,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:39,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681890999559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:39,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681890999563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:39,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681890999566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,570 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419c4b9f63e13224b5d9b0d7dc4f2788432 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c4b9f63e13224b5d9b0d7dc4f2788432 2023-04-19 07:55:39,576 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=567, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/78b07e4a9dd64da2acd8a5d876bbf2e4 2023-04-19 07:55:39,667 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:39,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:39,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:39,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:39,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:39,672 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:39,672 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:39,672 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:39,672 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:39,672 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,672 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:39,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681890999667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:39,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681890999668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:39,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681890999670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:39,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681890999671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,676 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:39,676 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:39,676 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,676 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:39,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:39,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:39,677 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:39,677 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:39,678 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:39,678 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:39,678 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:39,678 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:39,678 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:39,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,679 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:39,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:39,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:39,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a70a5cd[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:39,681 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:39,682 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,683 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,683 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:39,683 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:39,683 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,683 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:39,683 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:39,684 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:39,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:39,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:39,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:39,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:39,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:39,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:39,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:39,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:39,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:39,687 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:39,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:39,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:39,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:39,687 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:39,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:39,690 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:39,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:39,690 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:39,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,691 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:39,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,693 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,693 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:39,693 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,693 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:39,693 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:39,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:39,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:39,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:39,693 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:39,693 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,694 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,694 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,694 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:39,694 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:39,694 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:39,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:39,794 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:39,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:39,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:39,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:39,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:39,799 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:39,799 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:39,799 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:39,799 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:39,799 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,799 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,801 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,801 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:39,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:39,801 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,801 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:39,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:39,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:39,810 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:39,810 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:39,810 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:39,810 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:39,810 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:39,810 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:39,810 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,810 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:39,810 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:39,813 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:39,813 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:39,813 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,813 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:39,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:39,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:39,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e859ba[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:39,814 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:39,815 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,816 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:39,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:39,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:39,817 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,817 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:39,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:39,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:39,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:39,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:39,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:39,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:39,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:39,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:39,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:39,820 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:39,820 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:39,821 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:39,821 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:39,821 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:39,826 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:39,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:39,826 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,827 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,827 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:39,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:39,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,828 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:39,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,834 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:39,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:39,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:39,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:39,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:39,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:39,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:39,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:39,834 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:39,835 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:39,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,835 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,835 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,835 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:39,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681890999875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:39,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681890999876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:39,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681890999876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:39,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681890999876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,935 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:39,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:39,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:39,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:39,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:39,943 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:39,943 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:39,943 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:39,944 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:39,944 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,944 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,945 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:39,945 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:39,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:39,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,946 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,946 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:39,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:39,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:39,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:39,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:39,947 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:39,947 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:39,947 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:39,947 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:39,951 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:39,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:39,952 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:39,952 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:39,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:39,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:39,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:39,953 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:39,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4feaa835[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:39,953 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,954 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:39,954 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,954 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:39,956 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:39,956 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:39,956 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:39,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:39,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:39,959 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:39,960 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:39,960 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:39,960 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:39,960 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:39,961 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,961 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:39,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,962 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:39,963 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:39,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,964 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,967 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,967 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,967 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:39,967 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:39,967 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:39,967 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:39,968 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:39,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:39,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:39,968 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:39,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:39,968 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:39,968 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:39,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:39,968 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:39,968 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:39,987 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/a481a77e53e341318641bf9a370618e8 2023-04-19 07:55:40,005 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/288f12be1c99406bb5e46ad7afc868c8 2023-04-19 07:55:40,008 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/78b07e4a9dd64da2acd8a5d876bbf2e4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/78b07e4a9dd64da2acd8a5d876bbf2e4 2023-04-19 07:55:40,012 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/78b07e4a9dd64da2acd8a5d876bbf2e4, entries=250, sequenceid=567, filesize=39.2 K 2023-04-19 07:55:40,013 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/a481a77e53e341318641bf9a370618e8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/a481a77e53e341318641bf9a370618e8 2023-04-19 07:55:40,017 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/a481a77e53e341318641bf9a370618e8, entries=150, sequenceid=567, filesize=11.9 K 2023-04-19 07:55:40,018 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/288f12be1c99406bb5e46ad7afc868c8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/288f12be1c99406bb5e46ad7afc868c8 2023-04-19 07:55:40,025 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/288f12be1c99406bb5e46ad7afc868c8, entries=150, sequenceid=567, filesize=11.9 K 2023-04-19 07:55:40,029 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~149.83 KB/153430, heapSize ~417.20 KB/427216, currentSize=33.54 KB/34350 for a8342afdd5fe71a59bc1e541995ded31 in 474ms, sequenceid=567, compaction requested=true 2023-04-19 07:55:40,029 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:40,029 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:40,029 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:40,029 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:55:40,029 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 07:55:40,030 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:55:40,031 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111580 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:55:40,031 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/A is initiating minor compaction (all files) 2023-04-19 07:55:40,031 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/A in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:40,031 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/3d3709008af8459389393d9aa0daedc1, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/ebef4ca79f124216b0afafaf57800f20, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/46861bc650134480b411cf04ac7f1331, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/78b07e4a9dd64da2acd8a5d876bbf2e4] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=109.0 K 2023-04-19 07:55:40,031 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44220 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:55:40,031 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/B is initiating minor compaction (all files) 2023-04-19 07:55:40,031 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/B in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:40,031 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/b32b11dcfd2c448187be39ec1c6f8230, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/fe563a19ed24470990018ba68ff7a98c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/1949da7159ca4e148fa72b2513158f44, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/a481a77e53e341318641bf9a370618e8] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=43.2 K 2023-04-19 07:55:40,032 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 3d3709008af8459389393d9aa0daedc1, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681890936569 2023-04-19 07:55:40,032 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting b32b11dcfd2c448187be39ec1c6f8230, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681890936569 2023-04-19 07:55:40,032 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting ebef4ca79f124216b0afafaf57800f20, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681890936732 2023-04-19 07:55:40,032 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting fe563a19ed24470990018ba68ff7a98c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681890936732 2023-04-19 07:55:40,033 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 46861bc650134480b411cf04ac7f1331, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681890938635 2023-04-19 07:55:40,033 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 1949da7159ca4e148fa72b2513158f44, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681890938669 2023-04-19 07:55:40,033 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 78b07e4a9dd64da2acd8a5d876bbf2e4, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1681890939534 2023-04-19 07:55:40,033 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting a481a77e53e341318641bf9a370618e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1681890939534 2023-04-19 07:55:40,055 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#B#compaction#355 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:40,057 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#A#compaction#354 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:40,068 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:40,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:40,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:40,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:40,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:40,081 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:40,081 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:40,081 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:40,081 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:40,081 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:40,081 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:40,083 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:40,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:40,084 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:40,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:40,084 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,084 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:40,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:40,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:40,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:40,087 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:40,087 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:40,087 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:40,090 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:40,090 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:40,090 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:40,091 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:40,091 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/482e1994650249a586ec6395f4abc956 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/482e1994650249a586ec6395f4abc956 2023-04-19 07:55:40,095 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/B of a8342afdd5fe71a59bc1e541995ded31 into 482e1994650249a586ec6395f4abc956(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:40,095 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:40,095 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/B, priority=12, startTime=1681890940029; duration=0sec 2023-04-19 07:55:40,095 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:40,095 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:55:40,096 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-19 07:55:40,097 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:40,097 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:40,097 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:40,097 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:40,097 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:40,097 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:40,098 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44220 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:55:40,098 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/C is initiating minor compaction (all files) 2023-04-19 07:55:40,098 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/C in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:40,098 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d1e7db8ddd5843dbaabb71bdc0c6604e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/ff81b9d62b1a4b788f5e0ba65e73ded4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/0b11afc3f6484c52a9f3afc5497676d0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/288f12be1c99406bb5e46ad7afc868c8] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=43.2 K 2023-04-19 07:55:40,098 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting d1e7db8ddd5843dbaabb71bdc0c6604e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1681890936569 2023-04-19 07:55:40,100 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting ff81b9d62b1a4b788f5e0ba65e73ded4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681890936732 2023-04-19 07:55:40,100 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 0b11afc3f6484c52a9f3afc5497676d0, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681890938669 2023-04-19 07:55:40,101 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 288f12be1c99406bb5e46ad7afc868c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1681890939534 2023-04-19 07:55:40,129 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#C#compaction#357 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:40,136 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/659bf8a4673d475ebe37e6b36eac9770 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/659bf8a4673d475ebe37e6b36eac9770 2023-04-19 07:55:40,137 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304198a220bab8b6341b896e96351449021ed to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198a220bab8b6341b896e96351449021ed 2023-04-19 07:55:40,142 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/A of a8342afdd5fe71a59bc1e541995ded31 into 659bf8a4673d475ebe37e6b36eac9770(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:40,142 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:40,142 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/A, priority=12, startTime=1681890940029; duration=0sec 2023-04-19 07:55:40,142 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:40,143 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/8a355d81fa7345eda512bfdd46c9be75 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/8a355d81fa7345eda512bfdd46c9be75 2023-04-19 07:55:40,149 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/C of a8342afdd5fe71a59bc1e541995ded31 into 8a355d81fa7345eda512bfdd46c9be75(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:40,149 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:40,149 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/C, priority=12, startTime=1681890940029; duration=0sec 2023-04-19 07:55:40,149 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:40,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:40,193 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:40,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890940206, max timestamp=1681890940215Num uniques 859; 2023-04-19 07:55:40,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:55:40,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890940206, max timestamp=1681890940215Num uniques 859; 2023-04-19 07:55:40,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:55:40,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890940206, max timestamp=1681890940215Num uniques 859; 2023-04-19 07:55:40,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:55:40,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891000268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891000269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891000275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891000276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891000380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891000380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891000383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891000386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,544 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=576, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/ce986cc2a3b74fc88b50783592b4f278 2023-04-19 07:55:40,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891000588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891000588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891000589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891000591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891000893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891000893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891000895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:40,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891000895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:40,966 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=576 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/da56692f60754b879ab2a709c7e4e8ec 2023-04-19 07:55:41,082 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-19 07:55:41,112 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/edd34196153246fd897db9e892abbbf3, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/85d94892b2014b978e44718c02987121, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/88e90b35fb5247c78eca522a3d2dfbaf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/c4401770e22a4a0bbea48ebb6c7cd76a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0cab1c7be0294f568e6326530570a641, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/1514a2d9cf554ad68991c8ace074310f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/653557f8394f479198a58b2435fc1a3b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/414135528ad44e0d9534bd5391496eab, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0750e6d7cba64363a827cbc25b0e30f7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/1d85990923264f53b5142b928769c610, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/84b128705ac441d19bc2ea876d5b4ad9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/7fe6824f8f694e6397c6f6c0efab578d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/595a70d3cc0e48019c7fdf5af7882aba, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/20e8e740b83c493e922670bca8ec9947, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/4a8e3be4087b44ef83fd3c63b6becadd, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/3d3709008af8459389393d9aa0daedc1, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/ebef4ca79f124216b0afafaf57800f20, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/46861bc650134480b411cf04ac7f1331, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/78b07e4a9dd64da2acd8a5d876bbf2e4] to archive 2023-04-19 07:55:41,113 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:55:41,115 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/edd34196153246fd897db9e892abbbf3 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/edd34196153246fd897db9e892abbbf3 2023-04-19 07:55:41,117 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/85d94892b2014b978e44718c02987121 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/85d94892b2014b978e44718c02987121 2023-04-19 07:55:41,118 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/88e90b35fb5247c78eca522a3d2dfbaf to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/88e90b35fb5247c78eca522a3d2dfbaf 2023-04-19 07:55:41,119 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/c4401770e22a4a0bbea48ebb6c7cd76a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/c4401770e22a4a0bbea48ebb6c7cd76a 2023-04-19 07:55:41,120 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0cab1c7be0294f568e6326530570a641 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0cab1c7be0294f568e6326530570a641 2023-04-19 07:55:41,121 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/1514a2d9cf554ad68991c8ace074310f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/1514a2d9cf554ad68991c8ace074310f 2023-04-19 07:55:41,122 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/653557f8394f479198a58b2435fc1a3b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/653557f8394f479198a58b2435fc1a3b 2023-04-19 07:55:41,123 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/414135528ad44e0d9534bd5391496eab to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/414135528ad44e0d9534bd5391496eab 2023-04-19 07:55:41,125 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0750e6d7cba64363a827cbc25b0e30f7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0750e6d7cba64363a827cbc25b0e30f7 2023-04-19 07:55:41,126 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/1d85990923264f53b5142b928769c610 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/1d85990923264f53b5142b928769c610 2023-04-19 07:55:41,127 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/84b128705ac441d19bc2ea876d5b4ad9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/84b128705ac441d19bc2ea876d5b4ad9 2023-04-19 07:55:41,128 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/7fe6824f8f694e6397c6f6c0efab578d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/7fe6824f8f694e6397c6f6c0efab578d 2023-04-19 07:55:41,129 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/595a70d3cc0e48019c7fdf5af7882aba to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/595a70d3cc0e48019c7fdf5af7882aba 2023-04-19 07:55:41,130 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/20e8e740b83c493e922670bca8ec9947 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/20e8e740b83c493e922670bca8ec9947 2023-04-19 07:55:41,131 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/4a8e3be4087b44ef83fd3c63b6becadd to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/4a8e3be4087b44ef83fd3c63b6becadd 2023-04-19 07:55:41,132 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/3d3709008af8459389393d9aa0daedc1 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/3d3709008af8459389393d9aa0daedc1 2023-04-19 07:55:41,133 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/ebef4ca79f124216b0afafaf57800f20 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/ebef4ca79f124216b0afafaf57800f20 2023-04-19 07:55:41,137 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/46861bc650134480b411cf04ac7f1331 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/46861bc650134480b411cf04ac7f1331 2023-04-19 07:55:41,138 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/78b07e4a9dd64da2acd8a5d876bbf2e4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/78b07e4a9dd64da2acd8a5d876bbf2e4 2023-04-19 07:55:41,161 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/0051e3e1dd9e453d869cba288b28023f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f3aae6c9714d4616a78b0e3a38938b71, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2acafd61e3c84f3a9a7879409d57fc4b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/c968cfcfd986498d916e4173dd0a4cb5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/141b751147c646339dbac5f178132572, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/ac08b91f7a4d481f9afef1a5a18830a3, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/fac58c4fa0e847978c94cb15134b7381, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/4151d857cc36448cb83b220a8a4eefcf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2f4c719e937f49dabbc6412642d903f5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f37a7e34b38b440dafa9871fb5783ec7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/d6a1e8d9e60d413daa34a3209df52808, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/5ffb657c120e4c8b9f8b62362cbcd738, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2ef6d9baabc34bd3a903dc52e1e91780, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/19695ef5dba94efbaff425d849886ff5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/b32b11dcfd2c448187be39ec1c6f8230, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/1e3d09bb12264bd19feb90255b56f90a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/fe563a19ed24470990018ba68ff7a98c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/1949da7159ca4e148fa72b2513158f44, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/a481a77e53e341318641bf9a370618e8] to archive 2023-04-19 07:55:41,163 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:55:41,165 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/0051e3e1dd9e453d869cba288b28023f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/0051e3e1dd9e453d869cba288b28023f 2023-04-19 07:55:41,167 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f3aae6c9714d4616a78b0e3a38938b71 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f3aae6c9714d4616a78b0e3a38938b71 2023-04-19 07:55:41,168 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2acafd61e3c84f3a9a7879409d57fc4b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2acafd61e3c84f3a9a7879409d57fc4b 2023-04-19 07:55:41,169 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/c968cfcfd986498d916e4173dd0a4cb5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/c968cfcfd986498d916e4173dd0a4cb5 2023-04-19 07:55:41,172 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/141b751147c646339dbac5f178132572 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/141b751147c646339dbac5f178132572 2023-04-19 07:55:41,173 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/ac08b91f7a4d481f9afef1a5a18830a3 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/ac08b91f7a4d481f9afef1a5a18830a3 2023-04-19 07:55:41,174 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/fac58c4fa0e847978c94cb15134b7381 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/fac58c4fa0e847978c94cb15134b7381 2023-04-19 07:55:41,177 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/4151d857cc36448cb83b220a8a4eefcf to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/4151d857cc36448cb83b220a8a4eefcf 2023-04-19 07:55:41,182 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2f4c719e937f49dabbc6412642d903f5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2f4c719e937f49dabbc6412642d903f5 2023-04-19 07:55:41,189 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f37a7e34b38b440dafa9871fb5783ec7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f37a7e34b38b440dafa9871fb5783ec7 2023-04-19 07:55:41,192 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/d6a1e8d9e60d413daa34a3209df52808 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/d6a1e8d9e60d413daa34a3209df52808 2023-04-19 07:55:41,194 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/5ffb657c120e4c8b9f8b62362cbcd738 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/5ffb657c120e4c8b9f8b62362cbcd738 2023-04-19 07:55:41,195 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2ef6d9baabc34bd3a903dc52e1e91780 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/2ef6d9baabc34bd3a903dc52e1e91780 2023-04-19 07:55:41,196 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/19695ef5dba94efbaff425d849886ff5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/19695ef5dba94efbaff425d849886ff5 2023-04-19 07:55:41,209 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/b32b11dcfd2c448187be39ec1c6f8230 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/b32b11dcfd2c448187be39ec1c6f8230 2023-04-19 07:55:41,210 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/1e3d09bb12264bd19feb90255b56f90a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/1e3d09bb12264bd19feb90255b56f90a 2023-04-19 07:55:41,212 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/fe563a19ed24470990018ba68ff7a98c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/fe563a19ed24470990018ba68ff7a98c 2023-04-19 07:55:41,213 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/1949da7159ca4e148fa72b2513158f44 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/1949da7159ca4e148fa72b2513158f44 2023-04-19 07:55:41,214 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/a481a77e53e341318641bf9a370618e8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/a481a77e53e341318641bf9a370618e8 2023-04-19 07:55:41,220 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/00c5cd1086564da99c59becbc657bcd9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/33e8e1fc2921498f9aa6cfdb37c81047, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/baf6e8e767804ef4b1de2cada7444128, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3104641bf90a40d5a697a797227013c4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/79d6d41e177b4386a1b307f29ce94804, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7045629c307e4a0aac04e7a4176b02af, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/652aec2a0e1a4b3ba9e0d3eb3e8f778b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/025400c0f2024e3f9830a61a3620e99f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/79ec77abf22b42db93184969492c59af, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/168b2ad4ed6b44919d2f3daf1f80ffc7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d32fc42fd5184b698207d8ec181a3006, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/801e2626826d41c3a6938979614385ea, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7eb38b2b70974df8ab2f4bc0bfc01eea, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d183a255299a4c8688579f77faf59d48, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d1e7db8ddd5843dbaabb71bdc0c6604e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3da258ce40f74355a05279be076fb9fd, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/ff81b9d62b1a4b788f5e0ba65e73ded4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/0b11afc3f6484c52a9f3afc5497676d0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/288f12be1c99406bb5e46ad7afc868c8] to archive 2023-04-19 07:55:41,221 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:55:41,222 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/00c5cd1086564da99c59becbc657bcd9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/00c5cd1086564da99c59becbc657bcd9 2023-04-19 07:55:41,224 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/33e8e1fc2921498f9aa6cfdb37c81047 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/33e8e1fc2921498f9aa6cfdb37c81047 2023-04-19 07:55:41,225 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/baf6e8e767804ef4b1de2cada7444128 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/baf6e8e767804ef4b1de2cada7444128 2023-04-19 07:55:41,226 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3104641bf90a40d5a697a797227013c4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3104641bf90a40d5a697a797227013c4 2023-04-19 07:55:41,227 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/79d6d41e177b4386a1b307f29ce94804 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/79d6d41e177b4386a1b307f29ce94804 2023-04-19 07:55:41,228 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7045629c307e4a0aac04e7a4176b02af to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7045629c307e4a0aac04e7a4176b02af 2023-04-19 07:55:41,229 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/652aec2a0e1a4b3ba9e0d3eb3e8f778b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/652aec2a0e1a4b3ba9e0d3eb3e8f778b 2023-04-19 07:55:41,230 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/025400c0f2024e3f9830a61a3620e99f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/025400c0f2024e3f9830a61a3620e99f 2023-04-19 07:55:41,231 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/79ec77abf22b42db93184969492c59af to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/79ec77abf22b42db93184969492c59af 2023-04-19 07:55:41,232 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/168b2ad4ed6b44919d2f3daf1f80ffc7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/168b2ad4ed6b44919d2f3daf1f80ffc7 2023-04-19 07:55:41,236 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d32fc42fd5184b698207d8ec181a3006 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d32fc42fd5184b698207d8ec181a3006 2023-04-19 07:55:41,237 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/801e2626826d41c3a6938979614385ea to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/801e2626826d41c3a6938979614385ea 2023-04-19 07:55:41,238 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7eb38b2b70974df8ab2f4bc0bfc01eea to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7eb38b2b70974df8ab2f4bc0bfc01eea 2023-04-19 07:55:41,239 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d183a255299a4c8688579f77faf59d48 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d183a255299a4c8688579f77faf59d48 2023-04-19 07:55:41,240 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d1e7db8ddd5843dbaabb71bdc0c6604e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/d1e7db8ddd5843dbaabb71bdc0c6604e 2023-04-19 07:55:41,241 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3da258ce40f74355a05279be076fb9fd to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3da258ce40f74355a05279be076fb9fd 2023-04-19 07:55:41,244 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/ff81b9d62b1a4b788f5e0ba65e73ded4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/ff81b9d62b1a4b788f5e0ba65e73ded4 2023-04-19 07:55:41,245 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/0b11afc3f6484c52a9f3afc5497676d0 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/0b11afc3f6484c52a9f3afc5497676d0 2023-04-19 07:55:41,246 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/288f12be1c99406bb5e46ad7afc868c8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/288f12be1c99406bb5e46ad7afc868c8 2023-04-19 07:55:41,376 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=576 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/69ab416200fc4acb81c3a6541595fc64 2023-04-19 07:55:41,382 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/ce986cc2a3b74fc88b50783592b4f278 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/ce986cc2a3b74fc88b50783592b4f278 2023-04-19 07:55:41,391 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/ce986cc2a3b74fc88b50783592b4f278, entries=150, sequenceid=576, filesize=25.4 K 2023-04-19 07:55:41,392 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/da56692f60754b879ab2a709c7e4e8ec as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/da56692f60754b879ab2a709c7e4e8ec 2023-04-19 07:55:41,396 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/da56692f60754b879ab2a709c7e4e8ec, entries=150, sequenceid=576, filesize=11.9 K 2023-04-19 07:55:41,398 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/69ab416200fc4acb81c3a6541595fc64 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/69ab416200fc4acb81c3a6541595fc64 2023-04-19 07:55:41,402 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/69ab416200fc4acb81c3a6541595fc64, entries=150, sequenceid=576, filesize=11.9 K 2023-04-19 07:55:41,403 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for a8342afdd5fe71a59bc1e541995ded31 in 1307ms, sequenceid=576, compaction requested=false 2023-04-19 07:55:41,403 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:41,403 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:41,403 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:41,403 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:41,403 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,403 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:41,403 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:41,406 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,406 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,406 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,406 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:41,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:41,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@177fa2ff[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:41,408 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:41,408 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,409 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,409 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:41,409 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:41,409 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:41,409 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:41,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:41,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:41,411 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:41,411 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890940267, max timestamp=1681890941409Num uniques 1118; 2023-04-19 07:55:41,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-19 07:55:41,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890940267, max timestamp=1681890941409Num uniques 1118; 2023-04-19 07:55:41,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890940267, max timestamp=1681890941409Num uniques 1118; 2023-04-19 07:55:41,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-19 07:55:41,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-19 07:55:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:41,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:41,417 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:41,417 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:41,417 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:41,417 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:41,417 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:41,419 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,419 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,419 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:41,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,420 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:41,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:41,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:41,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,421 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=53.67 KB heapSize=145.08 KB 2023-04-19 07:55:41,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,422 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,422 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:41,423 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:41,423 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:41,423 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:41,423 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:41,423 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:41,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,427 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:41,427 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:41,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:41,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:41,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:41,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:41,427 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,427 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:41,428 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:41,428 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:41,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:41,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890941467, max timestamp=1681890941480Num uniques 859; 2023-04-19 07:55:41,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890941467, max timestamp=1681890941480Num uniques 859; 2023-04-19 07:55:41,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:55:41,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:55:41,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890941467, max timestamp=1681890941480Num uniques 859; 2023-04-19 07:55:41,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:55:41,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:41,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891001512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:41,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891001516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:41,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891001519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:41,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891001520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,528 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:41,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:41,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:41,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:41,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:41,531 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:41,531 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:41,531 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:41,531 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:41,531 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,532 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,532 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:41,532 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:41,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:41,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:41,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:41,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:41,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:41,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:41,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:41,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:41,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:41,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:41,534 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:41,534 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:41,534 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:41,535 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:41,535 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:41,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:41,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:41,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:41,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:41,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,536 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:41,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:41,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58ee2310[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:41,538 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:41,538 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,539 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,539 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:41,539 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:41,539 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,539 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:41,539 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:41,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,541 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:41,541 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,541 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:41,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:41,543 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:41,543 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:41,543 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:41,543 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:41,543 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:41,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,546 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:41,548 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,549 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,551 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,551 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,551 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,551 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:41,551 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:41,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:41,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:41,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:41,551 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,551 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:41,552 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,552 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:41,552 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:41,551 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:41,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:41,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:41,552 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:41,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891001621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:41,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891001625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:41,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891001626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:41,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891001628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,652 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:41,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:41,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:41,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:41,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:41,655 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:41,655 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:41,655 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:41,655 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:41,656 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,656 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,662 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:41,662 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:41,663 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,663 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:41,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:41,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:41,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:41,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:41,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:41,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:41,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:41,664 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:41,664 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:41,664 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:41,664 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:41,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:41,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:41,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,664 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:41,665 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:41,666 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,666 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,666 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:41,667 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:41,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b08843e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:41,667 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,668 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,669 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,669 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:41,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:41,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:41,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:41,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:41,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:41,670 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:41,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:41,673 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:41,673 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:41,673 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:41,673 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:41,673 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:41,676 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,676 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,676 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,676 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:41,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:41,679 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:41,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:41,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:41,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:41,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:41,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:41,680 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:41,680 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:41,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,680 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,680 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,680 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,680 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,680 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:41,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:41,780 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:41,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:41,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:41,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:41,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:41,783 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:41,783 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:41,783 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:41,783 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:41,783 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,783 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,784 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,784 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:41,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:41,785 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,785 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:41,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:41,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:41,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:41,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:41,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:41,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:41,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:41,786 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:41,786 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:41,786 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:41,786 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:41,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:41,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:41,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:41,786 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:41,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,788 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,788 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:41,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:41,789 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:41,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1393c6a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:41,789 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,792 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,792 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,792 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:41,792 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:41,792 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:41,792 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:41,793 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:41,793 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,793 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:41,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,793 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:41,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:41,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:41,795 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:41,796 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:41,796 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:41,796 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:41,796 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:41,797 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,797 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,797 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:41,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,797 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:41,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:41,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,798 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,800 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:41,800 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:41,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:41,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:41,801 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,801 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,801 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,801 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,801 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,801 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:41,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:41,801 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,801 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:41,801 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:41,801 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:41,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:41,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:41,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891001827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:41,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891001831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:41,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891001832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:41,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891001835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,865 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304196dcd13ee1709448594e72901408f0181 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304196dcd13ee1709448594e72901408f0181 2023-04-19 07:55:41,870 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=621, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/62e7b5582b6c4217b03118f9b7b918da 2023-04-19 07:55:41,883 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=621 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/f0fcf03b4066430195ed49f794e05f85 2023-04-19 07:55:41,901 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:41,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:41,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:41,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:41,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:41,922 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:41,922 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:41,922 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:41,922 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:41,923 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,923 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,923 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=621 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/7a7ba57509a448369478880309ff68a1 2023-04-19 07:55:41,925 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:41,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:41,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:41,925 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,926 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,926 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:41,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:41,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:41,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:41,927 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:41,927 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:41,927 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:41,928 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:41,928 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:41,928 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:41,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/62e7b5582b6c4217b03118f9b7b918da as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/62e7b5582b6c4217b03118f9b7b918da 2023-04-19 07:55:41,928 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:41,928 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:41,928 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:41,928 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:41,928 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:41,928 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,928 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:41,929 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:41,930 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,930 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,930 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,931 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:41,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:41,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b3b0ee8[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:41,933 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:41,933 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,934 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/62e7b5582b6c4217b03118f9b7b918da, entries=250, sequenceid=621, filesize=39.2 K 2023-04-19 07:55:41,934 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,934 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,935 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:41,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/f0fcf03b4066430195ed49f794e05f85 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f0fcf03b4066430195ed49f794e05f85 2023-04-19 07:55:41,935 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:41,935 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:41,935 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:41,936 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:41,936 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:41,936 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:41,936 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f0fcf03b4066430195ed49f794e05f85, entries=150, sequenceid=621, filesize=11.9 K 2023-04-19 07:55:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/7a7ba57509a448369478880309ff68a1 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7a7ba57509a448369478880309ff68a1 2023-04-19 07:55:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:41,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:41,940 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:41,940 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:41,940 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:41,940 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:41,940 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:41,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7a7ba57509a448369478880309ff68a1, entries=150, sequenceid=621, filesize=11.9 K 2023-04-19 07:55:41,948 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,949 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:41,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,949 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:41,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:41,949 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~164.95 KB/168912, currentSize=127.47 KB/130530 for a8342afdd5fe71a59bc1e541995ded31 in 528ms, sequenceid=621, compaction requested=true 2023-04-19 07:55:41,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:41,950 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:41,950 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:41,950 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 07:55:41,950 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:41,950 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:41,950 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,951 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:41,951 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/B is initiating minor compaction (all files) 2023-04-19 07:55:41,952 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/B in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:41,952 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/482e1994650249a586ec6395f4abc956, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/da56692f60754b879ab2a709c7e4e8ec, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f0fcf03b4066430195ed49f794e05f85] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=36.3 K 2023-04-19 07:55:41,952 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92870 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:41,952 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/A is initiating minor compaction (all files) 2023-04-19 07:55:41,952 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/A in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:41,952 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/659bf8a4673d475ebe37e6b36eac9770, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/ce986cc2a3b74fc88b50783592b4f278, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/62e7b5582b6c4217b03118f9b7b918da] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=90.7 K 2023-04-19 07:55:41,952 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 482e1994650249a586ec6395f4abc956, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1681890939534 2023-04-19 07:55:41,953 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 659bf8a4673d475ebe37e6b36eac9770, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1681890939534 2023-04-19 07:55:41,953 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting da56692f60754b879ab2a709c7e4e8ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1681890939557 2023-04-19 07:55:41,953 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting ce986cc2a3b74fc88b50783592b4f278, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1681890939557 2023-04-19 07:55:41,953 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting f0fcf03b4066430195ed49f794e05f85, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1681890941417 2023-04-19 07:55:41,954 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 62e7b5582b6c4217b03118f9b7b918da, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1681890940267 2023-04-19 07:55:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,954 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:41,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:41,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:41,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:41,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:41,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:41,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:41,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:41,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:41,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:41,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:41,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:41,958 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:41,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:41,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:41,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,958 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:41,958 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:41,958 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:41,961 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#B#compaction#364 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:41,968 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#A#compaction#363 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:42,058 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:42,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:42,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:42,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:42,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:42,062 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:42,062 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:42,062 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:42,062 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:42,062 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:42,063 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:42,068 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:42,068 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:42,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:42,068 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,069 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:42,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:42,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:42,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:42,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:42,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:42,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:42,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:42,070 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:42,070 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:42,070 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:42,070 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:42,070 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:42,070 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:42,070 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:42,070 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-19 07:55:42,071 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:42,071 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:42,071 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:42,071 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:42,071 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:42,071 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:42,108 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419337d44cc83ae44568868faed0965c799 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419337d44cc83ae44568868faed0965c799 2023-04-19 07:55:42,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:42,142 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:42,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891002154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891002154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891002157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891002158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891002264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891002264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891002271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891002271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,374 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/186090f11d564371a4db1364ae30cb9f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/186090f11d564371a4db1364ae30cb9f 2023-04-19 07:55:42,377 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/B of a8342afdd5fe71a59bc1e541995ded31 into 186090f11d564371a4db1364ae30cb9f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:42,378 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:42,378 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/B, priority=13, startTime=1681890941950; duration=0sec 2023-04-19 07:55:42,378 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:42,378 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:55:42,379 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:55:42,379 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/C is initiating minor compaction (all files) 2023-04-19 07:55:42,379 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/C in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:42,379 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/8a355d81fa7345eda512bfdd46c9be75, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/69ab416200fc4acb81c3a6541595fc64, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7a7ba57509a448369478880309ff68a1] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=36.3 K 2023-04-19 07:55:42,380 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 8a355d81fa7345eda512bfdd46c9be75, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1681890939534 2023-04-19 07:55:42,380 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 69ab416200fc4acb81c3a6541595fc64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1681890939557 2023-04-19 07:55:42,388 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 7a7ba57509a448369478880309ff68a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1681890941417 2023-04-19 07:55:42,409 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#C#compaction#366 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:42,414 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/0122421bc0f747e7a599f8002f0bae0c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0122421bc0f747e7a599f8002f0bae0c 2023-04-19 07:55:42,419 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/A of a8342afdd5fe71a59bc1e541995ded31 into 0122421bc0f747e7a599f8002f0bae0c(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:42,419 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:42,419 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/A, priority=13, startTime=1681890941950; duration=0sec 2023-04-19 07:55:42,420 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:42,431 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/38aeb8d1192d4235bc880024206d1689 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/38aeb8d1192d4235bc880024206d1689 2023-04-19 07:55:42,435 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/C of a8342afdd5fe71a59bc1e541995ded31 into 38aeb8d1192d4235bc880024206d1689(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:42,435 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:42,435 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/C, priority=13, startTime=1681890941950; duration=0sec 2023-04-19 07:55:42,435 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:42,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891002476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891002477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891002478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891002479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,514 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=657, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/bfd870bc08674904b784ee86f1551598 2023-04-19 07:55:42,534 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/18c4f6d1626346e89de18c9dd2b54e91 2023-04-19 07:55:42,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47030 deadline: 1681891002758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,762 DEBUG [Thread-2221] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18231 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:55:42,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891002783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891002784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891002784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:42,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891002791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:42,978 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/3eea55ce226e45b98ecc2ecc101cb517 2023-04-19 07:55:42,996 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/bfd870bc08674904b784ee86f1551598 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/bfd870bc08674904b784ee86f1551598 2023-04-19 07:55:43,000 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/bfd870bc08674904b784ee86f1551598, entries=150, sequenceid=657, filesize=25.4 K 2023-04-19 07:55:43,002 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/18c4f6d1626346e89de18c9dd2b54e91 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/18c4f6d1626346e89de18c9dd2b54e91 2023-04-19 07:55:43,005 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/18c4f6d1626346e89de18c9dd2b54e91, entries=150, sequenceid=657, filesize=11.9 K 2023-04-19 07:55:43,006 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/3eea55ce226e45b98ecc2ecc101cb517 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3eea55ce226e45b98ecc2ecc101cb517 2023-04-19 07:55:43,010 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3eea55ce226e45b98ecc2ecc101cb517, entries=150, sequenceid=657, filesize=11.9 K 2023-04-19 07:55:43,011 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for a8342afdd5fe71a59bc1e541995ded31 in 941ms, sequenceid=657, compaction requested=false 2023-04-19 07:55:43,011 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:43,011 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:43,011 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:43,011 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:43,011 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,012 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:43,012 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:43,014 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:43,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:43,015 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,015 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,015 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:43,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:43,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:43,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:43,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42e4563c[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:43,019 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:43,020 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,021 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,021 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,021 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:43,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:43,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:43,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:43,022 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:43,022 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:43,022 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:43,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:43,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:43,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:43,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:43,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:43,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:43,025 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:43,025 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:43,026 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:43,026 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:43,026 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:43,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:43,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:43,028 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:43,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:43,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,031 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:43,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:43,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:43,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:43,040 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:43,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:43,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:43,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:43,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:43,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:43,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:43,040 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,041 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:43,041 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:43,041 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,041 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:43,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:43,141 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:43,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:43,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:43,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:43,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:43,152 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:43,152 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:43,152 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:43,152 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:43,152 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,153 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:43,154 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:43,154 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:43,154 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,154 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:43,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:43,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:43,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:43,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:43,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:43,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:43,156 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:43,156 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:43,156 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:43,156 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:43,156 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:43,156 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:43,156 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:43,157 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 07:55:43,157 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:43,157 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:43,157 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:43,157 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:43,157 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:43,157 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:43,185 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419a2b2c9ec831946fe9c41d0921bb13548 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419a2b2c9ec831946fe9c41d0921bb13548 2023-04-19 07:55:43,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:43,297 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:43,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890943321, max timestamp=1681890943339Num uniques 859; 2023-04-19 07:55:43,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890943321, max timestamp=1681890943339Num uniques 859; 2023-04-19 07:55:43,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:55:43,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890943321, max timestamp=1681890943339Num uniques 859; 2023-04-19 07:55:43,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:55:43,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:55:43,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:43,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891003371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:43,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891003374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:43,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891003380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:43,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891003385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:43,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891003486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:43,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891003487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:43,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891003487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:43,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891003497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,594 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=671, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/07a0a8854c0d43dfa1efc976279eaac8 2023-04-19 07:55:43,645 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=671 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/971f36672f494c4bbb54df70e363371c 2023-04-19 07:55:43,686 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=671 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/fc9bd8523d44404780506571dcd55a03 2023-04-19 07:55:43,691 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/07a0a8854c0d43dfa1efc976279eaac8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/07a0a8854c0d43dfa1efc976279eaac8 2023-04-19 07:55:43,698 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/07a0a8854c0d43dfa1efc976279eaac8, entries=150, sequenceid=671, filesize=25.4 K 2023-04-19 07:55:43,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:43,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891003693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:43,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891003693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,700 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/971f36672f494c4bbb54df70e363371c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/971f36672f494c4bbb54df70e363371c 2023-04-19 07:55:43,704 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/971f36672f494c4bbb54df70e363371c, entries=150, sequenceid=671, filesize=11.9 K 2023-04-19 07:55:43,704 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/fc9bd8523d44404780506571dcd55a03 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/fc9bd8523d44404780506571dcd55a03 2023-04-19 07:55:43,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:43,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891003694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:43,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891003709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,716 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/fc9bd8523d44404780506571dcd55a03, entries=150, sequenceid=671, filesize=11.9 K 2023-04-19 07:55:43,721 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a8342afdd5fe71a59bc1e541995ded31 in 565ms, sequenceid=671, compaction requested=true 2023-04-19 07:55:43,721 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:43,721 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:43,723 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,723 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:43,723 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,723 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:43,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:43,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d12032a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:43,725 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:43,725 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,726 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:43,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:43,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:43,727 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,727 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:43,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:43,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:43,728 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:43,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:43,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:43,731 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:43,731 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:43,731 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:43,731 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:43,731 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:43,733 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,733 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:43,733 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:43,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:43,734 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,735 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,737 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,737 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:43,737 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:43,737 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,737 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,737 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:43,737 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,737 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:43,737 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:43,737 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:43,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:43,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:43,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:43,737 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:43,737 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:43,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:43,838 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:43,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:43,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:43,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:43,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:43,852 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:43,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:43,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:43,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:43,853 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,853 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:43,854 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:43,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:43,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:43,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:43,855 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,856 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:43,856 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:43,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:43,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:43,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:43,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:43,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:43,864 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:43,864 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:43,864 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:43,864 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. started... 2023-04-19 07:55:43,864 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-19 07:55:43,864 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:43,865 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:43,865 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:43,865 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:43,865 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:43,865 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:43,924 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419c67cb34e00894061ae1bd621160d1fa3 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c67cb34e00894061ae1bd621160d1fa3 2023-04-19 07:55:43,944 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=705, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/95dfb94e739d4dc484a3404adf3c8183 2023-04-19 07:55:44,011 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. as already flushing 2023-04-19 07:55:44,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:44,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:44,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891004023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:44,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891004024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:44,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891004031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:44,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891004036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:44,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891004138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:44,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891004138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:44,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891004139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:44,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891004147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:44,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46972 deadline: 1681891004351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:44,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46994 deadline: 1681891004351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:44,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46946 deadline: 1681891004351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,366 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=705 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/33b039dc22614d7d9b56273f77bd9394 2023-04-19 07:55:44,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:44,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46996 deadline: 1681891004358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,393 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=705 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/cda610664f6349c6b6a35a23b8d5d660 2023-04-19 07:55:44,398 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/95dfb94e739d4dc484a3404adf3c8183 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/95dfb94e739d4dc484a3404adf3c8183 2023-04-19 07:55:44,411 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/95dfb94e739d4dc484a3404adf3c8183, entries=150, sequenceid=705, filesize=25.4 K 2023-04-19 07:55:44,411 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/33b039dc22614d7d9b56273f77bd9394 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/33b039dc22614d7d9b56273f77bd9394 2023-04-19 07:55:44,416 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/33b039dc22614d7d9b56273f77bd9394, entries=150, sequenceid=705, filesize=11.9 K 2023-04-19 07:55:44,416 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/cda610664f6349c6b6a35a23b8d5d660 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/cda610664f6349c6b6a35a23b8d5d660 2023-04-19 07:55:44,419 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/cda610664f6349c6b6a35a23b8d5d660, entries=150, sequenceid=705, filesize=11.9 K 2023-04-19 07:55:44,421 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for a8342afdd5fe71a59bc1e541995ded31 in 557ms, sequenceid=705, compaction requested=true 2023-04-19 07:55:44,421 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:44,421 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:44,421 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:44,421 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:44,421 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,422 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:44,422 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:44,424 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:44,424 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:44,424 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:44,424 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:44,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:44,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:44,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:44,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5500a68e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:44,426 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:44,426 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:44,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:44,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:44,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:44,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:44,428 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:44,428 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,428 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:44,431 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:44,431 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:44,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:44,431 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:44,431 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:44,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:44,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:44,434 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:44,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:44,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:44,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:44,434 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:44,436 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:44,436 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:44,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:44,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:44,436 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:44,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:44,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:44,437 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:44,442 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:44,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,447 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:44,448 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:44,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:44,448 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,448 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:44,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:44,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:44,448 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:44,448 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:44,448 DEBUG [Thread-2227] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:44,448 DEBUG [Thread-2227] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:44,448 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:44,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:44,448 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:44,448 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:44,449 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:44,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:44,452 DEBUG [Thread-2232] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10377b50 to 127.0.0.1:58087 2023-04-19 07:55:44,452 DEBUG [Thread-2232] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:44,458 DEBUG [Thread-2236] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a35a28d to 127.0.0.1:58087 2023-04-19 07:55:44,458 DEBUG [Thread-2236] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:44,460 DEBUG [Thread-2228] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3620ee35 to 127.0.0.1:58087 2023-04-19 07:55:44,460 DEBUG [Thread-2228] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:44,463 DEBUG [Thread-2230] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:58087 2023-04-19 07:55:44,463 DEBUG [Thread-2230] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:44,466 DEBUG [Thread-2234] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:58087 2023-04-19 07:55:44,467 DEBUG [Thread-2234] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:44,548 DEBUG [Thread-2227] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:44,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:44,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:44,668 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-19 07:55:44,668 DEBUG [Thread-2225] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x419b69bf to 127.0.0.1:58087 2023-04-19 07:55:44,668 DEBUG [Thread-2219] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41507d16 to 127.0.0.1:58087 2023-04-19 07:55:44,668 DEBUG [Thread-2225] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:44,668 DEBUG [Thread-2223] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c8d927d to 127.0.0.1:58087 2023-04-19 07:55:44,668 DEBUG [Thread-2223] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:44,668 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:44,668 DEBUG [Thread-2219] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:44,668 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:44,669 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:44,669 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:44,669 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:44,669 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:44,678 DEBUG [Thread-2217] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4473f45c to 127.0.0.1:58087 2023-04-19 07:55:44,678 DEBUG [Thread-2217] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:45,086 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041984100df82e9346d8805a45865dc698cc to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041984100df82e9346d8805a45865dc698cc 2023-04-19 07:55:45,094 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=719, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/b44942ef770e41209310601369d30021 2023-04-19 07:55:45,103 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/c2be38ae2dfa4ab0bf0ad83bda7de785 2023-04-19 07:55:45,113 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=719 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/204f96ff08b34304b63fc538ee20b77c 2023-04-19 07:55:45,116 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/b44942ef770e41209310601369d30021 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/b44942ef770e41209310601369d30021 2023-04-19 07:55:45,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/b44942ef770e41209310601369d30021, entries=150, sequenceid=719, filesize=25.4 K 2023-04-19 07:55:45,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/c2be38ae2dfa4ab0bf0ad83bda7de785 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/c2be38ae2dfa4ab0bf0ad83bda7de785 2023-04-19 07:55:45,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/c2be38ae2dfa4ab0bf0ad83bda7de785, entries=150, sequenceid=719, filesize=11.9 K 2023-04-19 07:55:45,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/204f96ff08b34304b63fc538ee20b77c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/204f96ff08b34304b63fc538ee20b77c 2023-04-19 07:55:45,129 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/204f96ff08b34304b63fc538ee20b77c, entries=150, sequenceid=719, filesize=11.9 K 2023-04-19 07:55:45,130 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=6.71 KB/6870 for a8342afdd5fe71a59bc1e541995ded31 in 462ms, sequenceid=719, compaction requested=true 2023-04-19 07:55:45,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:45,130 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:55:45,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:45,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:45,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 07:55:45,130 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:55:45,131 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130333 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:55:45,132 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/A is initiating minor compaction (all files) 2023-04-19 07:55:45,132 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/A in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:45,132 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0122421bc0f747e7a599f8002f0bae0c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/bfd870bc08674904b784ee86f1551598, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/07a0a8854c0d43dfa1efc976279eaac8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/95dfb94e739d4dc484a3404adf3c8183, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/b44942ef770e41209310601369d30021] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=127.3 K 2023-04-19 07:55:45,132 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 0122421bc0f747e7a599f8002f0bae0c, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1681890941417 2023-04-19 07:55:45,132 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61033 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:55:45,132 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/B is initiating minor compaction (all files) 2023-04-19 07:55:45,133 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/B in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:45,133 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/186090f11d564371a4db1364ae30cb9f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/18c4f6d1626346e89de18c9dd2b54e91, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/971f36672f494c4bbb54df70e363371c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/33b039dc22614d7d9b56273f77bd9394, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/c2be38ae2dfa4ab0bf0ad83bda7de785] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=59.6 K 2023-04-19 07:55:45,133 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting bfd870bc08674904b784ee86f1551598, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681890941510 2023-04-19 07:55:45,133 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 186090f11d564371a4db1364ae30cb9f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1681890941417 2023-04-19 07:55:45,133 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 07a0a8854c0d43dfa1efc976279eaac8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1681890942152 2023-04-19 07:55:45,134 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 18c4f6d1626346e89de18c9dd2b54e91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681890941510 2023-04-19 07:55:45,134 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 95dfb94e739d4dc484a3404adf3c8183, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=705, earliestPutTs=1681890943350 2023-04-19 07:55:45,134 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 971f36672f494c4bbb54df70e363371c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1681890942152 2023-04-19 07:55:45,134 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting b44942ef770e41209310601369d30021, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1681890944035 2023-04-19 07:55:45,134 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 33b039dc22614d7d9b56273f77bd9394, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=705, earliestPutTs=1681890943350 2023-04-19 07:55:45,134 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting c2be38ae2dfa4ab0bf0ad83bda7de785, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1681890944035 2023-04-19 07:55:45,144 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#B#compaction#379 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:45,150 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#A#compaction#378 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:45,161 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/6752f520f8f744518cda50b4c6f7998a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/6752f520f8f744518cda50b4c6f7998a 2023-04-19 07:55:45,165 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/B of a8342afdd5fe71a59bc1e541995ded31 into 6752f520f8f744518cda50b4c6f7998a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:45,166 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:45,166 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/B, priority=11, startTime=1681890945130; duration=0sec 2023-04-19 07:55:45,166 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:45,166 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:55:45,167 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61033 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:55:45,167 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): a8342afdd5fe71a59bc1e541995ded31/C is initiating minor compaction (all files) 2023-04-19 07:55:45,167 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8342afdd5fe71a59bc1e541995ded31/C in TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:45,168 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/38aeb8d1192d4235bc880024206d1689, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3eea55ce226e45b98ecc2ecc101cb517, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/fc9bd8523d44404780506571dcd55a03, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/cda610664f6349c6b6a35a23b8d5d660, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/204f96ff08b34304b63fc538ee20b77c] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp, totalSize=59.6 K 2023-04-19 07:55:45,171 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 38aeb8d1192d4235bc880024206d1689, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1681890941417 2023-04-19 07:55:45,171 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 3eea55ce226e45b98ecc2ecc101cb517, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681890941510 2023-04-19 07:55:45,172 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting fc9bd8523d44404780506571dcd55a03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=671, earliestPutTs=1681890942152 2023-04-19 07:55:45,172 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting cda610664f6349c6b6a35a23b8d5d660, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=705, earliestPutTs=1681890943350 2023-04-19 07:55:45,172 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 204f96ff08b34304b63fc538ee20b77c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=719, earliestPutTs=1681890944035 2023-04-19 07:55:45,182 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): a8342afdd5fe71a59bc1e541995ded31#C#compaction#380 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:45,191 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/4b60776aa9b143efbaf0e2af76e3371d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/4b60776aa9b143efbaf0e2af76e3371d 2023-04-19 07:55:45,195 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/C of a8342afdd5fe71a59bc1e541995ded31 into 4b60776aa9b143efbaf0e2af76e3371d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:45,195 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:45,195 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/C, priority=11, startTime=1681890945130; duration=0sec 2023-04-19 07:55:45,195 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:45,576 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/c24121e9a4bb466a88f8c39f414ac2a2 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/c24121e9a4bb466a88f8c39f414ac2a2 2023-04-19 07:55:45,580 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a8342afdd5fe71a59bc1e541995ded31/A of a8342afdd5fe71a59bc1e541995ded31 into c24121e9a4bb466a88f8c39f414ac2a2(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:55:45,581 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:45,581 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31., storeName=a8342afdd5fe71a59bc1e541995ded31/A, priority=11, startTime=1681890945130; duration=0sec 2023-04-19 07:55:45,581 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:52,828 DEBUG [Thread-2221] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49b5bd0d to 127.0.0.1:58087 2023-04-19 07:55:52,828 DEBUG [Thread-2221] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:52,828 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-19 07:55:52,828 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 162 2023-04-19 07:55:52,828 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 161 2023-04-19 07:55:52,828 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 8 2023-04-19 07:55:52,828 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 157 2023-04-19 07:55:52,828 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 145 2023-04-19 07:55:52,828 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-19 07:55:52,828 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-19 07:55:52,829 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(402): scanned 1627 2023-04-19 07:55:52,829 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(403): verified 4881 rows 2023-04-19 07:55:52,829 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(402): scanned 1634 2023-04-19 07:55:52,829 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(403): verified 4898 rows 2023-04-19 07:55:52,829 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(402): scanned 1618 2023-04-19 07:55:52,829 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(403): verified 4852 rows 2023-04-19 07:55:52,829 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(402): scanned 1625 2023-04-19 07:55:52,829 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(403): verified 4875 rows 2023-04-19 07:55:52,829 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(402): scanned 1630 2023-04-19 07:55:52,829 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(403): verified 4890 rows 2023-04-19 07:55:52,829 INFO [Listener at localhost/42517] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-19 07:55:52,829 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0279325e to 127.0.0.1:58087 2023-04-19 07:55:52,829 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:55:52,833 INFO [Listener at localhost/42517] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-19 07:55:52,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-19 07:55:52,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:52,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-19 07:55:52,837 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890952837"}]},"ts":"1681890952837"} 2023-04-19 07:55:52,838 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-19 07:55:52,839 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-19 07:55:52,840 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8342afdd5fe71a59bc1e541995ded31, UNASSIGN}] 2023-04-19 07:55:52,840 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8342afdd5fe71a59bc1e541995ded31, UNASSIGN 2023-04-19 07:55:52,841 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=a8342afdd5fe71a59bc1e541995ded31, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:52,841 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890952841"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890952841"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890952841"}]},"ts":"1681890952841"} 2023-04-19 07:55:52,842 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:55:52,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-19 07:55:52,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:52,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a8342afdd5fe71a59bc1e541995ded31, disabling compactions & flushes 2023-04-19 07:55:52,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:52,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:52,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. after waiting 0 ms 2023-04-19 07:55:52,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:52,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing a8342afdd5fe71a59bc1e541995ded31 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-19 07:55:52,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=A 2023-04-19 07:55:52,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:52,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=B 2023-04-19 07:55:52,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:52,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a8342afdd5fe71a59bc1e541995ded31, store=C 2023-04-19 07:55:52,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:53,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-19 07:55:53,407 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304194da8db62327a4dd395822969771928a9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194da8db62327a4dd395822969771928a9 2023-04-19 07:55:53,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-19 07:55:53,811 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=727, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/0a0d941b828045ba8b7019ffe3ca3ba0 2023-04-19 07:55:53,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-19 07:55:54,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/796178424484400580fbab950d71c1d2 2023-04-19 07:55:54,629 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/b5af745d5ca5430295d7a15467a45ba1 2023-04-19 07:55:54,632 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/A/0a0d941b828045ba8b7019ffe3ca3ba0 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0a0d941b828045ba8b7019ffe3ca3ba0 2023-04-19 07:55:54,635 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0a0d941b828045ba8b7019ffe3ca3ba0, entries=100, sequenceid=727, filesize=18.6 K 2023-04-19 07:55:54,636 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/B/796178424484400580fbab950d71c1d2 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/796178424484400580fbab950d71c1d2 2023-04-19 07:55:54,639 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/796178424484400580fbab950d71c1d2, entries=100, sequenceid=727, filesize=9.5 K 2023-04-19 07:55:54,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/.tmp/C/b5af745d5ca5430295d7a15467a45ba1 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/b5af745d5ca5430295d7a15467a45ba1 2023-04-19 07:55:54,642 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/b5af745d5ca5430295d7a15467a45ba1, entries=100, sequenceid=727, filesize=9.5 K 2023-04-19 07:55:54,643 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for a8342afdd5fe71a59bc1e541995ded31 in 1648ms, sequenceid=727, compaction requested=false 2023-04-19 07:55:54,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/659bf8a4673d475ebe37e6b36eac9770, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/ce986cc2a3b74fc88b50783592b4f278, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/62e7b5582b6c4217b03118f9b7b918da, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0122421bc0f747e7a599f8002f0bae0c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/bfd870bc08674904b784ee86f1551598, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/07a0a8854c0d43dfa1efc976279eaac8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/95dfb94e739d4dc484a3404adf3c8183, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/b44942ef770e41209310601369d30021] to archive 2023-04-19 07:55:54,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:55:54,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/659bf8a4673d475ebe37e6b36eac9770 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/659bf8a4673d475ebe37e6b36eac9770 2023-04-19 07:55:54,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/ce986cc2a3b74fc88b50783592b4f278 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/ce986cc2a3b74fc88b50783592b4f278 2023-04-19 07:55:54,648 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/62e7b5582b6c4217b03118f9b7b918da to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/62e7b5582b6c4217b03118f9b7b918da 2023-04-19 07:55:54,649 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0122421bc0f747e7a599f8002f0bae0c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0122421bc0f747e7a599f8002f0bae0c 2023-04-19 07:55:54,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/bfd870bc08674904b784ee86f1551598 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/bfd870bc08674904b784ee86f1551598 2023-04-19 07:55:54,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/07a0a8854c0d43dfa1efc976279eaac8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/07a0a8854c0d43dfa1efc976279eaac8 2023-04-19 07:55:54,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/95dfb94e739d4dc484a3404adf3c8183 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/95dfb94e739d4dc484a3404adf3c8183 2023-04-19 07:55:54,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/b44942ef770e41209310601369d30021 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/b44942ef770e41209310601369d30021 2023-04-19 07:55:54,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/482e1994650249a586ec6395f4abc956, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/da56692f60754b879ab2a709c7e4e8ec, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/186090f11d564371a4db1364ae30cb9f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f0fcf03b4066430195ed49f794e05f85, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/18c4f6d1626346e89de18c9dd2b54e91, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/971f36672f494c4bbb54df70e363371c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/33b039dc22614d7d9b56273f77bd9394, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/c2be38ae2dfa4ab0bf0ad83bda7de785] to archive 2023-04-19 07:55:54,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:55:54,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/482e1994650249a586ec6395f4abc956 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/482e1994650249a586ec6395f4abc956 2023-04-19 07:55:54,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/da56692f60754b879ab2a709c7e4e8ec to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/da56692f60754b879ab2a709c7e4e8ec 2023-04-19 07:55:54,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/186090f11d564371a4db1364ae30cb9f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/186090f11d564371a4db1364ae30cb9f 2023-04-19 07:55:54,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f0fcf03b4066430195ed49f794e05f85 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/f0fcf03b4066430195ed49f794e05f85 2023-04-19 07:55:54,660 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/18c4f6d1626346e89de18c9dd2b54e91 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/18c4f6d1626346e89de18c9dd2b54e91 2023-04-19 07:55:54,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/971f36672f494c4bbb54df70e363371c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/971f36672f494c4bbb54df70e363371c 2023-04-19 07:55:54,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/33b039dc22614d7d9b56273f77bd9394 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/33b039dc22614d7d9b56273f77bd9394 2023-04-19 07:55:54,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/c2be38ae2dfa4ab0bf0ad83bda7de785 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/c2be38ae2dfa4ab0bf0ad83bda7de785 2023-04-19 07:55:54,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/8a355d81fa7345eda512bfdd46c9be75, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/69ab416200fc4acb81c3a6541595fc64, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/38aeb8d1192d4235bc880024206d1689, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7a7ba57509a448369478880309ff68a1, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3eea55ce226e45b98ecc2ecc101cb517, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/fc9bd8523d44404780506571dcd55a03, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/cda610664f6349c6b6a35a23b8d5d660, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/204f96ff08b34304b63fc538ee20b77c] to archive 2023-04-19 07:55:54,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:55:54,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/8a355d81fa7345eda512bfdd46c9be75 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/8a355d81fa7345eda512bfdd46c9be75 2023-04-19 07:55:54,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/69ab416200fc4acb81c3a6541595fc64 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/69ab416200fc4acb81c3a6541595fc64 2023-04-19 07:55:54,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/38aeb8d1192d4235bc880024206d1689 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/38aeb8d1192d4235bc880024206d1689 2023-04-19 07:55:54,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7a7ba57509a448369478880309ff68a1 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/7a7ba57509a448369478880309ff68a1 2023-04-19 07:55:54,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3eea55ce226e45b98ecc2ecc101cb517 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/3eea55ce226e45b98ecc2ecc101cb517 2023-04-19 07:55:54,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/fc9bd8523d44404780506571dcd55a03 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/fc9bd8523d44404780506571dcd55a03 2023-04-19 07:55:54,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/cda610664f6349c6b6a35a23b8d5d660 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/cda610664f6349c6b6a35a23b8d5d660 2023-04-19 07:55:54,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/204f96ff08b34304b63fc538ee20b77c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/204f96ff08b34304b63fc538ee20b77c 2023-04-19 07:55:54,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/recovered.edits/730.seqid, newMaxSeqId=730, maxSeqId=4 2023-04-19 07:55:54,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31. 2023-04-19 07:55:54,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a8342afdd5fe71a59bc1e541995ded31: 2023-04-19 07:55:54,678 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:54,679 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=a8342afdd5fe71a59bc1e541995ded31, regionState=CLOSED 2023-04-19 07:55:54,679 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890954679"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890954679"}]},"ts":"1681890954679"} 2023-04-19 07:55:54,681 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-19 07:55:54,681 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure a8342afdd5fe71a59bc1e541995ded31, server=jenkins-hbase4.apache.org,42359,1681890819191 in 1.8380 sec 2023-04-19 07:55:54,682 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-19 07:55:54,682 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a8342afdd5fe71a59bc1e541995ded31, UNASSIGN in 1.8410 sec 2023-04-19 07:55:54,683 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890954682"}]},"ts":"1681890954682"} 2023-04-19 07:55:54,683 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-19 07:55:54,685 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-19 07:55:54,686 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8520 sec 2023-04-19 07:55:54,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-19 07:55:54,941 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-19 07:55:54,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-19 07:55:54,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:54,944 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:54,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-19 07:55:54,945 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:54,950 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:54,952 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/recovered.edits] 2023-04-19 07:55:54,954 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0a0d941b828045ba8b7019ffe3ca3ba0 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/0a0d941b828045ba8b7019ffe3ca3ba0 2023-04-19 07:55:54,955 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/c24121e9a4bb466a88f8c39f414ac2a2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/A/c24121e9a4bb466a88f8c39f414ac2a2 2023-04-19 07:55:54,957 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/6752f520f8f744518cda50b4c6f7998a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/6752f520f8f744518cda50b4c6f7998a 2023-04-19 07:55:54,958 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/796178424484400580fbab950d71c1d2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/B/796178424484400580fbab950d71c1d2 2023-04-19 07:55:54,960 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/4b60776aa9b143efbaf0e2af76e3371d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/4b60776aa9b143efbaf0e2af76e3371d 2023-04-19 07:55:54,961 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/b5af745d5ca5430295d7a15467a45ba1 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/C/b5af745d5ca5430295d7a15467a45ba1 2023-04-19 07:55:54,963 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/recovered.edits/730.seqid to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31/recovered.edits/730.seqid 2023-04-19 07:55:54,964 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/a8342afdd5fe71a59bc1e541995ded31 2023-04-19 07:55:54,964 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 07:55:54,964 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-19 07:55:54,965 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-19 07:55:54,967 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304190bc9edd8a4d94a9882042c2cd2e0669b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304190bc9edd8a4d94a9882042c2cd2e0669b 2023-04-19 07:55:54,968 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041916fd41c0453949afabe874dcee56b957 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041916fd41c0453949afabe874dcee56b957 2023-04-19 07:55:54,969 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191e92f1c6acac47a491d94d7ceef6f34a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191e92f1c6acac47a491d94d7ceef6f34a 2023-04-19 07:55:54,970 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419337d44cc83ae44568868faed0965c799 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419337d44cc83ae44568868faed0965c799 2023-04-19 07:55:54,971 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194da8db62327a4dd395822969771928a9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194da8db62327a4dd395822969771928a9 2023-04-19 07:55:54,972 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195b39f4e25b034f3eb16b58b4c20b25e5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195b39f4e25b034f3eb16b58b4c20b25e5 2023-04-19 07:55:54,973 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304196dcd13ee1709448594e72901408f0181 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304196dcd13ee1709448594e72901408f0181 2023-04-19 07:55:54,974 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419703555bd367640388dea3faa451a2a9d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419703555bd367640388dea3faa451a2a9d 2023-04-19 07:55:54,974 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041984100df82e9346d8805a45865dc698cc to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041984100df82e9346d8805a45865dc698cc 2023-04-19 07:55:54,975 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419881f69e63b614ca3b1a49e46fe1f6803 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419881f69e63b614ca3b1a49e46fe1f6803 2023-04-19 07:55:54,976 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198a220bab8b6341b896e96351449021ed to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304198a220bab8b6341b896e96351449021ed 2023-04-19 07:55:54,977 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199f693f59288a450995a5d0fa997fd68d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199f693f59288a450995a5d0fa997fd68d 2023-04-19 07:55:54,978 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419a0fd7bd9b737474faceee7a0a85044a2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419a0fd7bd9b737474faceee7a0a85044a2 2023-04-19 07:55:54,979 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419a2b2c9ec831946fe9c41d0921bb13548 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419a2b2c9ec831946fe9c41d0921bb13548 2023-04-19 07:55:54,979 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b7897e49cb4a462abc8000953dd15637 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b7897e49cb4a462abc8000953dd15637 2023-04-19 07:55:54,980 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c4b9f63e13224b5d9b0d7dc4f2788432 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c4b9f63e13224b5d9b0d7dc4f2788432 2023-04-19 07:55:54,981 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c67cb34e00894061ae1bd621160d1fa3 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419c67cb34e00894061ae1bd621160d1fa3 2023-04-19 07:55:54,982 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419cb279b6a91c94d68bb342c45140e956d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419cb279b6a91c94d68bb342c45140e956d 2023-04-19 07:55:54,982 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d2db9c243d61454c8fba6771d8434907 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d2db9c243d61454c8fba6771d8434907 2023-04-19 07:55:54,983 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f0fc162207294263b4de67df1339a684 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f0fc162207294263b4de67df1339a684 2023-04-19 07:55:54,984 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f8ad4902460d4b599eb1118bf9944e68 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f8ad4902460d4b599eb1118bf9944e68 2023-04-19 07:55:54,985 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419fe3ee81ed4664e5a85e33795a6bac1ff to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419fe3ee81ed4664e5a85e33795a6bac1ff 2023-04-19 07:55:54,986 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-19 07:55:54,987 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:54,989 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-19 07:55:54,990 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-19 07:55:54,991 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:54,991 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-19 07:55:54,991 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681890954991"}]},"ts":"9223372036854775807"} 2023-04-19 07:55:54,993 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-19 07:55:54,993 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a8342afdd5fe71a59bc1e541995ded31, NAME => 'TestAcidGuarantees,,1681890921378.a8342afdd5fe71a59bc1e541995ded31.', STARTKEY => '', ENDKEY => ''}] 2023-04-19 07:55:54,993 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-19 07:55:54,993 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681890954993"}]},"ts":"9223372036854775807"} 2023-04-19 07:55:54,994 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-19 07:55:54,996 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:54,997 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 54 msec 2023-04-19 07:55:55,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-19 07:55:55,046 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-19 07:55:55,059 INFO [Listener at localhost/42517] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=226 (was 224) - Thread LEAK? -, OpenFileDescriptor=545 (was 514) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=462 (was 352) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6531 (was 6131) - AvailableMemoryMB LEAK? - 2023-04-19 07:55:55,071 INFO [Listener at localhost/42517] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=226, OpenFileDescriptor=545, MaxFileDescriptor=60000, SystemLoadAverage=462, ProcessCount=169, AvailableMemoryMB=6531 2023-04-19 07:55:55,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 07:55:55,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 07:55:55,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-19 07:55:55,076 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-19 07:55:55,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-19 07:55:55,076 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-19 07:55:55,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-19 07:55:55,077 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:55,078 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15 empty. 2023-04-19 07:55:55,078 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:55,078 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 07:55:55,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-19 07:55:55,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-19 07:55:55,485 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 07:55:55,487 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 199c6b7f1f3a9353227e2a51c7965b15, NAME => 'TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp 2023-04-19 07:55:55,494 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:55:55,494 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 199c6b7f1f3a9353227e2a51c7965b15, disabling compactions & flushes 2023-04-19 07:55:55,494 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:55,494 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:55,494 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. after waiting 0 ms 2023-04-19 07:55:55,494 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:55,494 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:55,494 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:55:55,496 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-19 07:55:55,497 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890955497"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890955497"}]},"ts":"1681890955497"} 2023-04-19 07:55:55,498 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-19 07:55:55,499 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-19 07:55:55,499 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890955499"}]},"ts":"1681890955499"} 2023-04-19 07:55:55,500 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-19 07:55:55,502 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=199c6b7f1f3a9353227e2a51c7965b15, ASSIGN}] 2023-04-19 07:55:55,504 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=199c6b7f1f3a9353227e2a51c7965b15, ASSIGN 2023-04-19 07:55:55,505 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=199c6b7f1f3a9353227e2a51c7965b15, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42359,1681890819191; forceNewPlan=false, retain=false 2023-04-19 07:55:55,656 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=199c6b7f1f3a9353227e2a51c7965b15, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:55,656 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890955656"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890955656"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890955656"}]},"ts":"1681890955656"} 2023-04-19 07:55:55,657 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:55:55,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-19 07:55:55,812 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:55,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 199c6b7f1f3a9353227e2a51c7965b15, NAME => 'TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.', STARTKEY => '', ENDKEY => ''} 2023-04-19 07:55:55,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:55,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:55:55,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:55,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:55,813 INFO [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:55,814 DEBUG [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A 2023-04-19 07:55:55,815 DEBUG [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A 2023-04-19 07:55:55,815 INFO [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:55:55,815 INFO [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 199c6b7f1f3a9353227e2a51c7965b15 columnFamilyName A 2023-04-19 07:55:55,815 INFO [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] regionserver.HStore(310): Store=199c6b7f1f3a9353227e2a51c7965b15/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:55:55,815 INFO [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:55,817 DEBUG [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B 2023-04-19 07:55:55,817 DEBUG [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B 2023-04-19 07:55:55,817 INFO [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:55:55,817 INFO [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 199c6b7f1f3a9353227e2a51c7965b15 columnFamilyName B 2023-04-19 07:55:55,817 INFO [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] regionserver.HStore(310): Store=199c6b7f1f3a9353227e2a51c7965b15/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:55:55,818 INFO [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:55,818 DEBUG [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C 2023-04-19 07:55:55,819 DEBUG [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C 2023-04-19 07:55:55,819 INFO [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:55:55,819 INFO [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 199c6b7f1f3a9353227e2a51c7965b15 columnFamilyName C 2023-04-19 07:55:55,819 INFO [StoreOpener-199c6b7f1f3a9353227e2a51c7965b15-1] regionserver.HStore(310): Store=199c6b7f1f3a9353227e2a51c7965b15/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:55:55,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:55,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:55,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:55,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 07:55:55,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:55,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 07:55:55,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 199c6b7f1f3a9353227e2a51c7965b15; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61576146, jitterRate=-0.08244392275810242}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 07:55:55,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:55:55,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., pid=49, masterSystemTime=1681890955809 2023-04-19 07:55:55,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:55,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:55,827 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=199c6b7f1f3a9353227e2a51c7965b15, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:55,827 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890955827"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681890955827"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681890955827"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681890955827"}]},"ts":"1681890955827"} 2023-04-19 07:55:55,829 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-19 07:55:55,829 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 in 171 msec 2023-04-19 07:55:55,830 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-19 07:55:55,830 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=199c6b7f1f3a9353227e2a51c7965b15, ASSIGN in 327 msec 2023-04-19 07:55:55,831 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-19 07:55:55,831 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890955831"}]},"ts":"1681890955831"} 2023-04-19 07:55:55,832 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-19 07:55:55,834 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-19 07:55:55,835 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 760 msec 2023-04-19 07:55:56,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-19 07:55:56,181 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-19 07:55:56,182 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x7bec116b to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:56,188 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@cf75c24, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:56,189 DEBUG [Listener at localhost/42517] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:56,191 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37864, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:56,192 DEBUG [Listener at localhost/42517] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-19 07:55:56,194 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52854, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-19 07:55:56,195 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x73eb7080 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:56,199 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@681750df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:56,199 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x2a987907 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:56,203 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2df63a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:56,204 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x37705401 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:56,207 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec943a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:56,208 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x15880ffb to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:56,213 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20ed9117, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:56,213 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x052203cb to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:56,217 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50002343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:56,217 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x5726fb8b to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:56,221 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6edd017, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:56,221 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x6f2b3c03 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:56,226 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fb168ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:56,227 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x70e2be04 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:56,233 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461f5f3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:56,233 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x067ea274 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:56,237 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63093e21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:56,237 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x78592df0 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:55:56,241 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b475d8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:55:56,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:56,248 DEBUG [hconnection-0x6463ddef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:56,252 DEBUG [hconnection-0x776af7e2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:56,253 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37890, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:56,256 DEBUG [hconnection-0x87d0c3a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:56,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:56,261 DEBUG [hconnection-0xd54f597-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:56,261 DEBUG [hconnection-0x81c4161-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:56,262 DEBUG [hconnection-0x697a9a20-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:56,262 DEBUG [hconnection-0x4d5b837b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:56,261 DEBUG [hconnection-0x28eadd82-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:56,262 DEBUG [hconnection-0x787290-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:56,262 DEBUG [hconnection-0x9a367cf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:55:56,262 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37878, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:56,262 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37896, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:56,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:56,263 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:56,263 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:56,263 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:56,263 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:56,263 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37912, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:56,264 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37928, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:56,264 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,264 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,269 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37918, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:56,269 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37902, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:56,269 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:56,270 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37932, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:56,270 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37924, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:56,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:56,270 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37936, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:55:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,276 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,276 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:56,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:56,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:56,280 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:56,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:56,284 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:56,284 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:55:56,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:56,292 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 07:55:56,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:55:56,293 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:56,293 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:55:56,293 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:56,293 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:55:56,293 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:56,293 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:55:56,293 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:56,293 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:56,293 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:56,293 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,293 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:56,293 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:56,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,297 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,297 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:56,297 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:56,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66ffae88[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:56,299 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:56,299 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,300 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,301 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:56,301 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:56,301 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,301 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:56,301 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:56,303 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:56,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:56,308 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:56,308 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:56,308 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:56,309 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:56,309 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:56,315 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,315 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:56,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,315 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,316 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891016314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891016316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891016316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891016309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681890956301, max timestamp=1681890956314Num uniques 859; 2023-04-19 07:55:56,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891016317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,318 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681890956307, max timestamp=1681890956314Num uniques 859; 2023-04-19 07:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681890956307, max timestamp=1681890956314Num uniques 859; 2023-04-19 07:55:56,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-19 07:55:56,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-19 07:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:56,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:56,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:56,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,321 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:56,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:56,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:56,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,321 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:56,321 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:56,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:56,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:56,337 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/c3bc5ba20cb94d43b5ebd34778713576 2023-04-19 07:55:56,376 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/04e08a792716461f8eea445466a58d44 2023-04-19 07:55:56,422 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:56,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:56,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:56,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891016435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891016433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891016436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891016437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891016437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:56,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:56,455 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:56,455 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:56,455 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:56,455 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:56,455 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,455 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,456 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:56,456 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:56,457 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,457 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:56,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:56,458 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:56,459 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:56,459 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:56,460 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:56,460 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:56,460 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:56,460 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:55:56,460 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:55:56,460 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:56,460 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:56,460 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:56,460 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,464 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:56,464 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:56,465 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,465 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:56,467 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ca51dd[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:56,467 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,468 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,468 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:56,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:56,471 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,471 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,471 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,471 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:56,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:56,473 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:56,473 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:56,473 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:56,473 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:56,473 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:56,475 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,475 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,475 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:56,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:56,475 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,477 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,479 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:56,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:56,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:56,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:56,479 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:56,479 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:56,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:56,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:56,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:56,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:56,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891016539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891016539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891016539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891016539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891016539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,580 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:56,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:56,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:56,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:56,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:56,588 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:56,588 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:56,588 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:56,588 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:56,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,593 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,593 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:56,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:56,594 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,594 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:56,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:56,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:56,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:56,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:56,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:56,601 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:56,601 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:55:56,601 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:55:56,601 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:56,603 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,604 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:56,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bcc197d[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:56,605 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:56,605 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,606 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,607 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:56,607 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:56,607 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:56,607 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,607 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:56,609 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,609 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:56,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:56,612 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:56,612 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:56,612 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:56,612 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:56,612 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:56,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:56,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:56,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,615 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,617 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:56,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:56,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:56,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:56,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:56,620 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:56,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:56,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:56,620 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:56,620 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:56,720 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:56,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:56,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:56,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:56,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:56,724 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:56,724 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:56,724 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:56,724 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:56,724 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,725 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,726 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,726 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:56,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:56,727 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,727 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:56,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:56,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:56,728 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:56,728 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:55:56,728 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:55:56,728 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,728 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:56,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:56,730 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,730 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,731 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,731 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:56,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ba8ea5c[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:56,733 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:56,733 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,734 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,734 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:56,734 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:56,734 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:56,734 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,734 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:56,735 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:56,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:56,737 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:56,737 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:56,737 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:56,737 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:56,737 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:56,740 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,740 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,740 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:56,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:56,741 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,742 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:56,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:56,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:56,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:56,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:56,746 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:56,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,746 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:56,747 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:56,747 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:56,747 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:56,747 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,747 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,747 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891016744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891016745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891016745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891016746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:56,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891016747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,842 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/c2028394db71437f8b778ae91de25013 2023-04-19 07:55:56,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/c3bc5ba20cb94d43b5ebd34778713576 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/c3bc5ba20cb94d43b5ebd34778713576 2023-04-19 07:55:56,847 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:56,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:56,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:56,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:56,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:56,850 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:56,850 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:56,850 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:56,851 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:56,851 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,851 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/c3bc5ba20cb94d43b5ebd34778713576, entries=150, sequenceid=13, filesize=11.6 K 2023-04-19 07:55:56,852 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:56,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:56,852 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/04e08a792716461f8eea445466a58d44 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/04e08a792716461f8eea445466a58d44 2023-04-19 07:55:56,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:56,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:56,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:56,854 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:56,854 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:56,854 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:56,854 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:56,854 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:56,854 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:56,854 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:55:56,854 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:55:56,854 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:56,854 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:56,854 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:56,854 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,855 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:56,855 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:56,856 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/04e08a792716461f8eea445466a58d44, entries=150, sequenceid=13, filesize=11.6 K 2023-04-19 07:55:56,856 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,856 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,857 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,857 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,858 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/c2028394db71437f8b778ae91de25013 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/c2028394db71437f8b778ae91de25013 2023-04-19 07:55:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:56,858 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:56,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4038944f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:56,858 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:56,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:56,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:56,860 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,860 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:56,862 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/c2028394db71437f8b778ae91de25013, entries=150, sequenceid=13, filesize=11.6 K 2023-04-19 07:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:56,862 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,863 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=117.74 KB/120562 for 199c6b7f1f3a9353227e2a51c7965b15 in 571ms, sequenceid=13, compaction requested=false 2023-04-19 07:55:56,863 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:55:56,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:56,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:56,866 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:56,866 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:56,866 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:56,866 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:56,866 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:56,868 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,868 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,868 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:56,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:56,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:56,868 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,870 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,872 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:56,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:56,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:56,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:56,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:56,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:56,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:56,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:56,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:56,873 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:56,873 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:56,873 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,973 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:56,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:56,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:56,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:56,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:56,976 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:56,976 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:56,976 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:56,976 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:56,977 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,977 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,978 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:56,978 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:56,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:56,978 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:56,978 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:56,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:56,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:56,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:56,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:56,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:56,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:56,980 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:56,980 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:56,980 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:55:56,980 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=117.74 KB heapSize=333.55 KB 2023-04-19 07:55:56,980 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:55:56,980 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:56,980 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:55:56,980 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:56,980 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:55:56,980 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:56,993 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/7b914d2a74ec439b9e4051e34661e8f9 2023-04-19 07:55:57,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:57,057 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:55:57,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891017060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891017061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891017066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891017066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891017068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891017172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891017172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891017172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891017173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891017173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891017376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891017376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891017376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891017379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891017380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,413 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=37.62 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/53d11f47046646c5990cc5edbf9f2bb8 2023-04-19 07:55:57,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891017678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891017679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891017680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891017686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:57,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891017686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,825 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=37.62 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/db6dcea247c94f0bb56efd243990c0fa 2023-04-19 07:55:57,829 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/7b914d2a74ec439b9e4051e34661e8f9 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/7b914d2a74ec439b9e4051e34661e8f9 2023-04-19 07:55:57,832 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/7b914d2a74ec439b9e4051e34661e8f9, entries=150, sequenceid=47, filesize=11.6 K 2023-04-19 07:55:57,833 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/53d11f47046646c5990cc5edbf9f2bb8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/53d11f47046646c5990cc5edbf9f2bb8 2023-04-19 07:55:57,837 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/53d11f47046646c5990cc5edbf9f2bb8, entries=150, sequenceid=47, filesize=11.6 K 2023-04-19 07:55:57,838 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/db6dcea247c94f0bb56efd243990c0fa as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/db6dcea247c94f0bb56efd243990c0fa 2023-04-19 07:55:57,847 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/db6dcea247c94f0bb56efd243990c0fa, entries=150, sequenceid=47, filesize=11.6 K 2023-04-19 07:55:57,848 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~117.74 KB/120562, heapSize ~333.50 KB/341504, currentSize=60.38 KB/61830 for 199c6b7f1f3a9353227e2a51c7965b15 in 868ms, sequenceid=47, compaction requested=false 2023-04-19 07:55:57,848 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:55:57,848 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:57,848 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,852 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:57,852 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:57,861 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:57,861 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:57,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:57,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14a8712c[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:57,863 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:57,863 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:57,864 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:57,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:57,864 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,864 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:57,865 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:57,865 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:57,879 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:57,879 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:57,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:57,882 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:57,882 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:57,883 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:57,883 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:57,883 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:57,885 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:57,885 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:57,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:57,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:57,886 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:57,887 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,888 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,892 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:57,892 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:57,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:57,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:57,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:57,892 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:57,892 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:57,892 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:57,892 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:57,893 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:57,893 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:57,893 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,893 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:57,893 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:57,893 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:57,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:57,992 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:57,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:57,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:57,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:57,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:57,996 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:57,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:57,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:57,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:57,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:57,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:57,997 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,997 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:57,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:57,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:57,998 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:57,998 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:57,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:57,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:57,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:57,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:57,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:57,999 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:57,999 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:57,999 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:57,999 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:57,999 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:57,999 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:57,999 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:55:58,000 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 07:55:58,000 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:55:58,000 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:58,000 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:55:58,000 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:58,000 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:55:58,000 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:55:58,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:58,187 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:55:58,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891018209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891018210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891018211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891018211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891018217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891018318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891018320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891018320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891018320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891018325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,414 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/0d1d458d450749e59ff0ef607f008a44 2023-04-19 07:55:58,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891018524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891018527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891018528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891018528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891018531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,825 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/87d68a94653b4b67b31b8c25fa776769 2023-04-19 07:55:58,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891018829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891018829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891018830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891018834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:58,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:58,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891018836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,236 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/f1ffa481ad2d496fa23003dc06e2079d 2023-04-19 07:55:59,240 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/0d1d458d450749e59ff0ef607f008a44 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/0d1d458d450749e59ff0ef607f008a44 2023-04-19 07:55:59,243 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/0d1d458d450749e59ff0ef607f008a44, entries=150, sequenceid=59, filesize=11.6 K 2023-04-19 07:55:59,244 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/87d68a94653b4b67b31b8c25fa776769 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/87d68a94653b4b67b31b8c25fa776769 2023-04-19 07:55:59,247 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/87d68a94653b4b67b31b8c25fa776769, entries=150, sequenceid=59, filesize=11.6 K 2023-04-19 07:55:59,248 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/f1ffa481ad2d496fa23003dc06e2079d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/f1ffa481ad2d496fa23003dc06e2079d 2023-04-19 07:55:59,251 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/f1ffa481ad2d496fa23003dc06e2079d, entries=150, sequenceid=59, filesize=11.6 K 2023-04-19 07:55:59,251 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 199c6b7f1f3a9353227e2a51c7965b15 in 1251ms, sequenceid=59, compaction requested=true 2023-04-19 07:55:59,251 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:55:59,251 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:59,251 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:59,252 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:59,252 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,252 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:59,252 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:59,253 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,253 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:59,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f448535[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:59,255 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:59,255 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,257 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,257 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:59,258 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,258 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:59,258 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:59,258 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:59,259 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,259 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:59,259 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:59,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,259 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:59,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:59,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:59,262 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:59,262 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:59,262 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:59,262 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:59,262 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:59,263 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,263 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,263 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:59,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:59,264 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,266 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,268 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,268 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,268 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,268 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,268 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,268 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:59,268 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:59,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:59,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:59,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:59,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:59,269 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:59,269 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,269 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,269 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:59,269 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:59,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:59,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:59,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:59,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890958209, max timestamp=1681890959339Num uniques 859; 2023-04-19 07:55:59,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:55:59,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890958209, max timestamp=1681890959339Num uniques 859; 2023-04-19 07:55:59,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890958209, max timestamp=1681890959339Num uniques 859; 2023-04-19 07:55:59,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:55:59,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:55:59,342 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=26.84 KB heapSize=66.23 KB 2023-04-19 07:55:59,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:55:59,343 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:59,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:55:59,343 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:59,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:55:59,343 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:59,369 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:59,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:59,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:59,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:59,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:59,372 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:59,372 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:59,372 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:59,372 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:59,373 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,373 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,376 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,376 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:59,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:59,376 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,376 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:59,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:59,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:59,378 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:59,378 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:55:59,378 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:55:59,378 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:59,379 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,379 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,380 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,380 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:59,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63226a8b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:59,381 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:59,381 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,383 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:59,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:59,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:59,383 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,383 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:59,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890959370, max timestamp=1681890959383Num uniques 859; 2023-04-19 07:55:59,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890959370, max timestamp=1681890959383Num uniques 859; 2023-04-19 07:55:59,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:55:59,385 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:59,385 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:59,385 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:59,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890959370, max timestamp=1681890959383Num uniques 859; 2023-04-19 07:55:59,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:55:59,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:55:59,385 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:59,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:59,390 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:59,390 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:59,390 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:59,390 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:59,390 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:59,392 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,392 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:59,392 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,392 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,394 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,395 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,395 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,395 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:59,395 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:59,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:59,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:59,395 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:59,395 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:59,395 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,396 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,396 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,396 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:59,396 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:59,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:59,396 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:59,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890959386, max timestamp=1681890959459Num uniques 1068; 2023-04-19 07:55:59,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:59,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-19 07:55:59,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891019459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890959386, max timestamp=1681890959459Num uniques 1068; 2023-04-19 07:55:59,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890959386, max timestamp=1681890959459Num uniques 1068; 2023-04-19 07:55:59,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-19 07:55:59,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-19 07:55:59,496 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:59,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:59,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:59,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:59,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:59,499 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:59,499 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:59,499 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:59,499 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:59,500 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,500 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,501 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:59,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:59,501 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,501 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,501 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:59,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:59,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:59,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:59,502 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:59,502 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:59,502 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:59,503 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:59,503 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:59,503 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:59,503 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:55:59,503 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:55:59,503 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:59,503 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:59,503 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:59,503 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,503 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:59,503 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:59,504 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,504 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,505 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,505 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:59,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:59,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:59,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67bc104c[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:59,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,508 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,508 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,509 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:59,508 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:59,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:59,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:59,515 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:59,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891019503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890959477, max timestamp=1681890959515Num uniques 1127; 2023-04-19 07:55:59,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-19 07:55:59,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890959477, max timestamp=1681890959515Num uniques 1127; 2023-04-19 07:55:59,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-19 07:55:59,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890959477, max timestamp=1681890959515Num uniques 1127; 2023-04-19 07:55:59,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-19 07:55:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:59,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:59,520 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:59,520 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:59,520 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:59,520 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:59,520 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:59,521 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,521 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,521 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:59,521 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,523 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,525 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:59,525 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,525 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:59,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:59,525 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:59,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:59,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:59,525 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,525 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,525 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,525 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,525 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,525 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,525 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:59,525 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:59,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:59,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681890959535, max timestamp=1681890959575Num uniques 1086; 2023-04-19 07:55:59,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681890959535, max timestamp=1681890959575Num uniques 1086; 2023-04-19 07:55:59,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-19 07:55:59,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-19 07:55:59,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681890959535, max timestamp=1681890959575Num uniques 1086; 2023-04-19 07:55:59,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-19 07:55:59,625 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:59,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:59,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:59,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:59,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:59,629 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:59,629 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:59,629 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:59,629 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:59,629 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,629 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,632 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:59,632 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:59,632 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,632 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:59,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:59,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:59,634 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:59,634 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:59,634 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:59,634 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:59,634 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:59,634 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:59,634 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:55:59,634 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:55:59,635 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:59,635 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:59,635 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:59,635 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,635 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:59,635 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:59,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:59,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891019625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:59,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891019626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:59,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891019633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681890959622, max timestamp=1681890959635Num uniques 1145; 2023-04-19 07:55:59,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-19 07:55:59,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681890959622, max timestamp=1681890959635Num uniques 1145; 2023-04-19 07:55:59,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-19 07:55:59,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681890959622, max timestamp=1681890959635Num uniques 1145; 2023-04-19 07:55:59,638 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-19 07:55:59,638 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,638 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:59,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30ca2c86[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:59,640 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:59,640 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,641 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,641 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:59,641 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:59,641 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,641 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:59,641 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:59,642 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:59,643 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:59,642 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,643 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:59,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:59,645 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:59,645 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:59,645 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:59,645 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:59,645 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:59,648 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,648 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,648 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:59,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,649 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:59,650 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,652 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,652 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:59,652 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:59,652 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:59,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:59,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:59,652 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:59,652 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,653 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,653 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,653 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,653 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,653 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:59,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:59,653 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:59,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:59,750 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/619f738c707b4adfb5b99a63e3bb2348 2023-04-19 07:55:59,753 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:59,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:59,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:59,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:59,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:59,757 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:59,757 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:59,757 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:59,757 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:59,758 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,758 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,759 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:59,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:59,759 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/3ae3f7b72852401e9ef7d1030e33bb23 2023-04-19 07:55:59,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:59,760 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,760 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:59,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:59,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:59,761 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:59,761 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:59,761 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:59,761 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:59,761 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:59,761 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:59,761 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:55:59,761 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:55:59,761 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:59,761 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:59,761 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:59,761 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,761 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:59,762 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:59,763 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,764 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,764 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,764 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:59,765 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:59,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b7c6a87[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:59,765 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,767 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:59,767 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,767 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:59,768 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:59,768 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:59,768 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,768 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:59,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:59,771 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:59,771 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:59,771 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:59,771 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:59,771 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:59,773 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,773 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,773 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:59,774 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,775 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:59,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891019738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:59,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891019743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,777 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681890959721, max timestamp=1681890959743Num uniques 1104; 2023-04-19 07:55:59,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-19 07:55:59,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681890959721, max timestamp=1681890959755Num uniques 1104; 2023-04-19 07:55:59,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-19 07:55:59,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681890959721, max timestamp=1681890959759Num uniques 1104; 2023-04-19 07:55:59,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-19 07:55:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,780 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,780 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:59,780 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:59,780 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:59,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:59,780 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:59,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:55:59,780 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,781 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,781 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:59,781 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:59,780 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:59,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:59,781 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,781 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,781 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/7200999589db4cf5a62dc439cf0df2aa 2023-04-19 07:55:59,785 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/619f738c707b4adfb5b99a63e3bb2348 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/619f738c707b4adfb5b99a63e3bb2348 2023-04-19 07:55:59,788 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/619f738c707b4adfb5b99a63e3bb2348, entries=250, sequenceid=81, filesize=16.2 K 2023-04-19 07:55:59,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/3ae3f7b72852401e9ef7d1030e33bb23 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/3ae3f7b72852401e9ef7d1030e33bb23 2023-04-19 07:55:59,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/3ae3f7b72852401e9ef7d1030e33bb23, entries=150, sequenceid=81, filesize=11.6 K 2023-04-19 07:55:59,792 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/7200999589db4cf5a62dc439cf0df2aa as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/7200999589db4cf5a62dc439cf0df2aa 2023-04-19 07:55:59,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/7200999589db4cf5a62dc439cf0df2aa, entries=150, sequenceid=81, filesize=11.6 K 2023-04-19 07:55:59,800 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=40.25 KB/41220 for 199c6b7f1f3a9353227e2a51c7965b15 in 458ms, sequenceid=81, compaction requested=true 2023-04-19 07:55:59,800 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:55:59,800 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:55:59,801 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:59,801 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:55:59,801 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:55:59,801 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:55:59,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:55:59,801 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=60.38 KB heapSize=166.31 KB 2023-04-19 07:55:59,801 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:55:59,801 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:59,801 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:55:59,801 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:59,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:55:59,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:55:59,802 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:55:59,802 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/B is initiating minor compaction (all files) 2023-04-19 07:55:59,802 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/B in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:59,802 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/04e08a792716461f8eea445466a58d44, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/53d11f47046646c5990cc5edbf9f2bb8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/87d68a94653b4b67b31b8c25fa776769, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/3ae3f7b72852401e9ef7d1030e33bb23] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=46.4 K 2023-04-19 07:55:59,802 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52224 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:55:59,803 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/A is initiating minor compaction (all files) 2023-04-19 07:55:59,803 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/A in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:59,803 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/c3bc5ba20cb94d43b5ebd34778713576, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/7b914d2a74ec439b9e4051e34661e8f9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/0d1d458d450749e59ff0ef607f008a44, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/619f738c707b4adfb5b99a63e3bb2348] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=51 K 2023-04-19 07:55:59,804 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 04e08a792716461f8eea445466a58d44, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681890956278 2023-04-19 07:55:59,804 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting c3bc5ba20cb94d43b5ebd34778713576, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681890956278 2023-04-19 07:55:59,804 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 7b914d2a74ec439b9e4051e34661e8f9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681890956434 2023-04-19 07:55:59,804 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 53d11f47046646c5990cc5edbf9f2bb8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681890956434 2023-04-19 07:55:59,805 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 87d68a94653b4b67b31b8c25fa776769, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681890957060 2023-04-19 07:55:59,805 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 0d1d458d450749e59ff0ef607f008a44, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681890957060 2023-04-19 07:55:59,805 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 3ae3f7b72852401e9ef7d1030e33bb23, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681890958209 2023-04-19 07:55:59,805 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 619f738c707b4adfb5b99a63e3bb2348, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681890958209 2023-04-19 07:55:59,819 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/4879c16199354307a3f446b0c7a124d4 2023-04-19 07:55:59,825 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#A#compaction#397 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:59,828 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#B#compaction#398 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:55:59,881 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:55:59,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:55:59,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:55:59,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:55:59,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:55:59,885 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:55:59,885 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:59,885 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:55:59,885 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:55:59,885 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,885 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,891 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,891 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:59,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:59,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:59,891 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,892 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:55:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:55:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:55:59,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:55:59,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:55:59,893 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:55:59,893 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:55:59,893 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:55:59,893 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:55:59,893 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:55:59,893 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:59,893 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:55:59,893 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:55:59,893 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:55:59,893 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:55:59,893 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:55:59,893 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,893 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:55:59,893 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:55:59,895 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,895 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,895 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,895 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:55:59,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:55:59,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b7a3351[Count = 0] remaining members to acquire global barrier 2023-04-19 07:55:59,898 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:55:59,898 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,899 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,899 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:55:59,899 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:55:59,899 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:55:59,899 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,899 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:55:59,900 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:55:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:55:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:55:59,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:55:59,902 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:55:59,902 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:55:59,902 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:55:59,902 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:55:59,902 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:55:59,905 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,905 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:59,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,906 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:55:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:55:59,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:59,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890959881, max timestamp=1681890959904Num uniques 859; 2023-04-19 07:55:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:55:59,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:55:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890959881, max timestamp=1681890959904Num uniques 859; 2023-04-19 07:55:59,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:55:59,906 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890959881, max timestamp=1681890959904Num uniques 859; 2023-04-19 07:55:59,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:55:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:55:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,909 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:55:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:55:59,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,913 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,913 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,913 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:55:59,913 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:55:59,913 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:55:59,913 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:55:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:55:59,913 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,913 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:55:59,913 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:55:59,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:55:59,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:55:59,914 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:55:59,914 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:55:59,914 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:55:59,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:55:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:55:59,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:59,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891019936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:59,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891019937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,948 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:59,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891019938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:59,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891019939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:55:59,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:55:59,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891019980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,014 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:00,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:00,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:00,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:00,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:00,017 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:00,018 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,018 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:00,018 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:00,018 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,018 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,019 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,019 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,019 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,020 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:00,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:00,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:00,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:00,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:00,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:00,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:00,021 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,021 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:00,021 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:00,021 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:00,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:00,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:00,021 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:00,023 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,023 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:00,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cc36e02[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:00,025 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:00,025 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,027 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:00,028 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:00,028 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:00,028 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,028 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:00,029 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,029 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:00,029 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,029 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:00,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:00,031 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:00,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:00,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:00,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:00,032 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:00,033 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,033 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,033 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,034 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,035 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:00,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:00,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,038 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:00,038 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:00,038 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:00,039 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891020042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891020047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891020050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891020050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,138 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:00,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:00,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:00,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:00,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:00,143 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:00,143 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,143 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:00,143 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:00,144 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,144 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,146 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,146 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,146 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,146 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:00,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:00,148 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,148 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:00,148 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:00,148 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:00,150 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,150 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:00,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d0cdf00[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:00,152 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:00,152 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,153 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,153 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:00,153 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:00,153 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,153 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:00,153 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:00,154 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:00,155 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,155 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:00,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:00,159 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:00,159 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:00,159 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:00,159 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:00,159 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:00,162 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,162 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,163 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,163 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,166 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:00,168 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:00,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:00,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,169 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:00,169 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:00,203 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-19 07:56:00,245 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/6372678b64a14a0694dbb6451f67a48e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/6372678b64a14a0694dbb6451f67a48e 2023-04-19 07:56:00,248 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/B of 199c6b7f1f3a9353227e2a51c7965b15 into 6372678b64a14a0694dbb6451f67a48e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:00,249 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:00,249 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/B, priority=12, startTime=1681890959801; duration=0sec 2023-04-19 07:56:00,249 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:00,249 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:56:00,250 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/2e097fa8064e4f92b645489d17538b28 2023-04-19 07:56:00,250 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:56:00,250 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/C is initiating minor compaction (all files) 2023-04-19 07:56:00,251 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/C in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,251 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/c2028394db71437f8b778ae91de25013, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/db6dcea247c94f0bb56efd243990c0fa, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/f1ffa481ad2d496fa23003dc06e2079d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/7200999589db4cf5a62dc439cf0df2aa] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=46.4 K 2023-04-19 07:56:00,251 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting c2028394db71437f8b778ae91de25013, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681890956278 2023-04-19 07:56:00,252 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting db6dcea247c94f0bb56efd243990c0fa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681890956434 2023-04-19 07:56:00,253 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting f1ffa481ad2d496fa23003dc06e2079d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681890957060 2023-04-19 07:56:00,253 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7200999589db4cf5a62dc439cf0df2aa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681890958209 2023-04-19 07:56:00,256 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/3ee656e5b5364abeb7d242c7c612fda5 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/3ee656e5b5364abeb7d242c7c612fda5 2023-04-19 07:56:00,262 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#C#compaction#401 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:00,263 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/A of 199c6b7f1f3a9353227e2a51c7965b15 into 3ee656e5b5364abeb7d242c7c612fda5(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:00,263 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:00,263 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/A, priority=12, startTime=1681890959800; duration=0sec 2023-04-19 07:56:00,263 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:00,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891020252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891020260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891020260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891020261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,269 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:00,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:00,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:00,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:00,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:00,272 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:00,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:00,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:00,273 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,273 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,274 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,274 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,274 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,274 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:00,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:00,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:00,280 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:00,280 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,280 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:00,284 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,284 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:00,284 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:00,284 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:00,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,286 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,287 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,287 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891020286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:00,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@752caa74[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:00,288 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:00,288 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,290 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,290 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:00,291 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:00,291 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:00,291 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,291 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:00,292 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,292 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:00,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:00,295 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:00,295 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:00,295 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:00,295 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:00,295 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:00,298 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,298 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,298 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,298 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,299 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,306 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,306 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,306 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,306 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,306 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,307 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:00,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:00,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:00,308 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:00,308 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:00,408 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:00,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:00,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:00,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:00,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:00,411 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:00,411 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,411 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:00,411 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:00,412 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,412 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,413 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,413 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,414 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,414 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:00,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:00,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:00,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:00,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:00,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:00,416 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,416 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:00,416 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:00,416 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:00,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:00,420 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f86ff3b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:00,422 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:00,422 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,424 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,424 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:00,424 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:00,424 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:00,424 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,424 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:00,425 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:00,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:00,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:00,428 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:00,428 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:00,428 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:00,428 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:00,428 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:00,430 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,430 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,430 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,430 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,433 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,435 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,435 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,435 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,435 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,435 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,435 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,435 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,435 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:00,435 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,435 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:00,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:00,436 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:00,436 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:00,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,536 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:00,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:00,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:00,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:00,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:00,539 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:00,539 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,539 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:00,539 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:00,540 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,540 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,541 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,542 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,542 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:00,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:00,543 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:00,543 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,543 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:00,543 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:00,543 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:00,543 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,543 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:00,544 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:00,544 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,544 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:00,544 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:00,544 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,544 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:00,544 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:00,545 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,545 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,546 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,546 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:00,547 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@383235b7[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:00,547 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,548 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:00,548 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,548 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:00,549 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:00,549 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,549 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,549 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:00,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:00,554 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:00,554 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:00,554 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:00,554 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:00,554 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:00,557 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,557 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,557 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,557 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,558 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,560 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:00,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:00,560 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,561 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,561 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,561 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:00,561 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:00,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:00,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891020569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891020569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891020569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891020569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,661 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:00,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:00,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:00,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:00,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:00,665 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:00,665 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,665 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:00,665 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:00,665 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,665 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/0ad3ad7b58f74506ba485a9444782f5c 2023-04-19 07:56:00,667 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,667 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,667 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,667 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:00,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:00,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:00,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:00,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:00,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:00,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:00,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:00,670 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,670 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:00,670 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:00,670 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:00,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:00,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,670 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/4879c16199354307a3f446b0c7a124d4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/4879c16199354307a3f446b0c7a124d4 2023-04-19 07:56:00,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:00,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:00,672 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,672 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,673 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,673 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:00,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23dd196e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:00,674 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:00,674 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/4879c16199354307a3f446b0c7a124d4, entries=150, sequenceid=193, filesize=11.8 K 2023-04-19 07:56:00,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/2e097fa8064e4f92b645489d17538b28 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2e097fa8064e4f92b645489d17538b28 2023-04-19 07:56:00,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2e097fa8064e4f92b645489d17538b28, entries=150, sequenceid=193, filesize=11.8 K 2023-04-19 07:56:00,684 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/0ad3ad7b58f74506ba485a9444782f5c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0ad3ad7b58f74506ba485a9444782f5c 2023-04-19 07:56:00,684 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,684 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:00,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:00,685 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:00,685 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:00,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,689 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/bd0bfecd35914c0597971f220325d632 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/bd0bfecd35914c0597971f220325d632 2023-04-19 07:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0ad3ad7b58f74506ba485a9444782f5c, entries=150, sequenceid=193, filesize=11.8 K 2023-04-19 07:56:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:00,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:00,693 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:00,693 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:00,693 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:00,693 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:00,693 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:00,693 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~166.27 KB/170256, currentSize=127.47 KB/130530 for 199c6b7f1f3a9353227e2a51c7965b15 in 892ms, sequenceid=193, compaction requested=false 2023-04-19 07:56:00,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:00,694 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,694 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,694 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,696 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/C of 199c6b7f1f3a9353227e2a51c7965b15 into bd0bfecd35914c0597971f220325d632(size=11.7 K), total size for store is 23.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,696 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,696 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:00,696 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/C, priority=12, startTime=1681890959801; duration=0sec 2023-04-19 07:56:00,696 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,698 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:00,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:00,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:00,701 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,701 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,701 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,701 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,701 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:00,701 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:00,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:00,795 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-19 07:56:00,795 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:00,795 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:00,795 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:00,795 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:00,795 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:00,795 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:00,802 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:00,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:00,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:00,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:00,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:00,805 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:00,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:00,806 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:00,806 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,806 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,811 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,811 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,811 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:00,811 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:00,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:00,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:00,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:00,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:00,816 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,816 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:00,816 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:00,816 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:00,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:00,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:00,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:00,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,818 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:00,820 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:00,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33b2686b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:00,820 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,823 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,823 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,823 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:00,823 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:00,823 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:00,823 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:00,827 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,828 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,828 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,828 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:00,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:00,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:00,833 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:00,833 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:00,833 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:00,833 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:00,833 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:00,835 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,835 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,835 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,835 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,837 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,839 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,839 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:00,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:00,840 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,841 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:00,841 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:00,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:00,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891020844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,941 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:00,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:00,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:00,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:00,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:00,945 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:00,946 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,946 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:00,946 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:00,946 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,946 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,947 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,947 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,948 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,948 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:00,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:00,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:00,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:00,949 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:00,949 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:00,949 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:00,949 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:00,949 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:00,949 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,949 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:00,950 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:00,950 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:00,950 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:00,950 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:00,950 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,950 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:00,950 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:00,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:00,952 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891020947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,952 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,954 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,954 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:00,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:00,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bc53601[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:00,954 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:00,954 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,955 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,956 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,956 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:00,956 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:00,956 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:00,956 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:00,957 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:00,957 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,957 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:00,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:00,961 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:00,961 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:00,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:00,961 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:00,961 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:00,961 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:00,962 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:00,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:00,962 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,962 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:00,963 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:00,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:00,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,964 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:00,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:00,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,968 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,968 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:00,968 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,968 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:00,968 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:00,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:00,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:00,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:00,968 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:00,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:00,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:00,968 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:00,969 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,969 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:00,969 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:00,969 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:00,969 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:00,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:01,069 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:01,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:01,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:01,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:01,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:01,072 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:01,072 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:01,072 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:01,073 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:01,073 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,073 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,074 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:01,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:01,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:01,075 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,075 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:01,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:01,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:01,076 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,076 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:01,076 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:01,076 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,076 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:01,077 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:01,078 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,078 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,078 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,078 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:01,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:01,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f32343[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:01,080 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:01,080 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,081 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,082 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,082 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:01,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,082 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:01,082 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:01,082 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:01,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:01,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891021079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:01,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891021079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,085 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:01,085 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:01,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:01,087 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:01,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:01,087 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:01,088 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:01,088 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:01,088 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:01,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891021082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:01,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891021082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,105 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,105 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,105 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:01,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,105 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,106 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,108 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:01,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:01,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:01,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,109 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:01,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:01,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:01,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:01,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:01,110 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:01,111 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:01,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:01,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891021154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/32fc60d3b49f4f09b10b7e136a87a6f7 2023-04-19 07:56:01,211 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:01,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:01,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:01,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:01,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:01,214 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:01,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:01,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:01,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:01,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,216 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,217 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:01,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:01,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:01,217 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,217 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,217 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:01,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:01,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:01,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:01,218 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:01,218 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:01,218 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:01,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:01,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:01,219 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,219 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:01,219 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:01,219 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:01,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:01,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:01,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:01,228 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,228 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,228 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,228 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:01,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@592e4e1a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:01,230 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:01,230 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,231 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,231 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:01,231 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:01,231 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:01,231 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,231 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:01,232 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,232 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:01,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:01,236 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:01,236 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:01,236 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:01,236 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:01,236 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:01,237 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,237 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,237 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:01,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,238 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,239 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,242 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,242 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,242 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,242 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,242 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:01,242 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:01,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:01,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:01,242 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,242 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,242 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:01,242 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:01,242 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:01,242 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:01,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:01,343 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:01,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:01,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:01,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:01,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:01,346 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:01,346 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:01,346 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:01,346 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:01,346 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,346 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:01,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:01,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:01,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:01,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:01,349 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:01,349 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:01,349 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:01,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:01,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:01,350 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,350 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:01,350 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:01,350 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:01,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:01,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:01,350 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:01,351 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,351 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,352 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,352 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:01,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@797f8464[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:01,353 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:01,353 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,354 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,354 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:01,354 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:01,355 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:01,355 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,355 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:01,357 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,357 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:01,357 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,357 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:01,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:01,359 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:01,359 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:01,359 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:01,359 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:01,359 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:01,361 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,361 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,362 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,362 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:01,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:01,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:01,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,363 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,365 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:01,365 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,365 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,365 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,365 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,365 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,365 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,365 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:01,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:01,365 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:01,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:01,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:01,365 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:01,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:01,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:01,366 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:01,366 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:01,466 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:01,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:01,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:01,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891021463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:01,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:01,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:01,470 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:01,470 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:01,470 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:01,470 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:01,470 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,470 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,471 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:01,471 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:01,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:01,472 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,472 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:01,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:01,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:01,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:01,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:01,473 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:01,473 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:01,473 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:01,473 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:01,473 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:01,473 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,473 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:01,473 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:01,474 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:01,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:01,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:01,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:01,476 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,476 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,477 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,477 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:01,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:01,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b18d074[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:01,478 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:01,478 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,479 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,480 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:01,480 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:01,480 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:01,480 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,480 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:01,481 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:01,481 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:01,481 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,481 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:01,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:01,488 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:01,488 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:01,488 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:01,488 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:01,488 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:01,490 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,490 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:01,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,490 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,491 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,492 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,495 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:01,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:01,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:01,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:01,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:01,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:01,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:01,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:01,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:01,495 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:01,495 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:01,595 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:01,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:01,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:01,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:01,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:01,599 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:01,599 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:01,599 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:01,599 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:01,600 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,600 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,601 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,601 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:01,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:01,601 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,601 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:01,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:01,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:01,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:01,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:01,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:01,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:01,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:01,603 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,603 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:01,603 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:01,603 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:01,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:01,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:01,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:01,605 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,605 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,605 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,606 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:01,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60fa1f6f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:01,607 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:01,607 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,608 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,609 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:01,609 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:01,609 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,609 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:01,609 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:01,613 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:01,613 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,613 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,614 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:01,620 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/acad54861c0d4f21b8c4189f91330dcc 2023-04-19 07:56:01,620 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:01,620 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:01,620 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:01,620 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:01,620 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:01,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:01,622 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,622 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,622 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,623 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:01,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:01,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:01,623 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,626 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:01,626 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:01,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:01,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:01,626 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,626 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,626 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,626 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,626 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,626 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,626 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,626 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:01,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:01,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:01,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,626 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:01,627 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:01,632 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/304e7b57d3244f4c881be1d68c3c378a 2023-04-19 07:56:01,636 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/32fc60d3b49f4f09b10b7e136a87a6f7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/32fc60d3b49f4f09b10b7e136a87a6f7 2023-04-19 07:56:01,641 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/32fc60d3b49f4f09b10b7e136a87a6f7, entries=200, sequenceid=231, filesize=14.1 K 2023-04-19 07:56:01,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/acad54861c0d4f21b8c4189f91330dcc as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/acad54861c0d4f21b8c4189f91330dcc 2023-04-19 07:56:01,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/acad54861c0d4f21b8c4189f91330dcc, entries=150, sequenceid=231, filesize=11.8 K 2023-04-19 07:56:01,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/304e7b57d3244f4c881be1d68c3c378a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/304e7b57d3244f4c881be1d68c3c378a 2023-04-19 07:56:01,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/304e7b57d3244f4c881be1d68c3c378a, entries=150, sequenceid=231, filesize=11.8 K 2023-04-19 07:56:01,649 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 199c6b7f1f3a9353227e2a51c7965b15 in 854ms, sequenceid=231, compaction requested=true 2023-04-19 07:56:01,649 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:01,649 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:01,649 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:01,649 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:01,649 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:01,649 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:01,650 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38486 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:01,650 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/A is initiating minor compaction (all files) 2023-04-19 07:56:01,650 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/A in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,651 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/3ee656e5b5364abeb7d242c7c612fda5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/4879c16199354307a3f446b0c7a124d4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/32fc60d3b49f4f09b10b7e136a87a6f7] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=37.6 K 2023-04-19 07:56:01,651 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 3ee656e5b5364abeb7d242c7c612fda5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681890958209 2023-04-19 07:56:01,651 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 4879c16199354307a3f446b0c7a124d4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681890959793 2023-04-19 07:56:01,651 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 32fc60d3b49f4f09b10b7e136a87a6f7, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681890959935 2023-04-19 07:56:01,652 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36096 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:01,652 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/B is initiating minor compaction (all files) 2023-04-19 07:56:01,652 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/B in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,652 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/6372678b64a14a0694dbb6451f67a48e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2e097fa8064e4f92b645489d17538b28, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/acad54861c0d4f21b8c4189f91330dcc] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=35.3 K 2023-04-19 07:56:01,652 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 6372678b64a14a0694dbb6451f67a48e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681890958209 2023-04-19 07:56:01,653 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 2e097fa8064e4f92b645489d17538b28, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681890959793 2023-04-19 07:56:01,653 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting acad54861c0d4f21b8c4189f91330dcc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681890959937 2023-04-19 07:56:01,658 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#A#compaction#405 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:01,665 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#B#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:01,686 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/69357012cc4c424aaa6cec4332778d58 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/69357012cc4c424aaa6cec4332778d58 2023-04-19 07:56:01,690 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/B of 199c6b7f1f3a9353227e2a51c7965b15 into 69357012cc4c424aaa6cec4332778d58(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:01,690 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:01,690 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/B, priority=13, startTime=1681890961649; duration=0sec 2023-04-19 07:56:01,690 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:01,690 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:01,691 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36096 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:01,691 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/C is initiating minor compaction (all files) 2023-04-19 07:56:01,691 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/C in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,691 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/bd0bfecd35914c0597971f220325d632, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0ad3ad7b58f74506ba485a9444782f5c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/304e7b57d3244f4c881be1d68c3c378a] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=35.3 K 2023-04-19 07:56:01,691 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting bd0bfecd35914c0597971f220325d632, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681890958209 2023-04-19 07:56:01,692 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 0ad3ad7b58f74506ba485a9444782f5c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681890959793 2023-04-19 07:56:01,693 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 304e7b57d3244f4c881be1d68c3c378a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681890959937 2023-04-19 07:56:01,698 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#C#compaction#407 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:01,727 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:01,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:01,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:01,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:01,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:01,730 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:01,730 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:01,730 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:01,730 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:01,731 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,731 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,732 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:01,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:01,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:01,732 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,733 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,733 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:01,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:01,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:01,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:01,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:01,734 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:01,734 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:01,734 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:01,735 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:01,735 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:01,735 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,735 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:01,735 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:56:01,735 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:01,735 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:01,735 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:01,735 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:01,735 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:01,735 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:01,750 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/914b73a88b1f436eacd250184ab831af 2023-04-19 07:56:01,761 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/3a47a84c53d94ded89fab4480be777d8 2023-04-19 07:56:01,774 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/b10e1361dd9b4b36a269c6079ff9affa 2023-04-19 07:56:01,778 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/914b73a88b1f436eacd250184ab831af as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/914b73a88b1f436eacd250184ab831af 2023-04-19 07:56:01,783 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/914b73a88b1f436eacd250184ab831af, entries=150, sequenceid=242, filesize=11.8 K 2023-04-19 07:56:01,783 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/3a47a84c53d94ded89fab4480be777d8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/3a47a84c53d94ded89fab4480be777d8 2023-04-19 07:56:01,788 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/3a47a84c53d94ded89fab4480be777d8, entries=150, sequenceid=242, filesize=11.8 K 2023-04-19 07:56:01,789 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/b10e1361dd9b4b36a269c6079ff9affa as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b10e1361dd9b4b36a269c6079ff9affa 2023-04-19 07:56:01,793 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b10e1361dd9b4b36a269c6079ff9affa, entries=150, sequenceid=242, filesize=11.8 K 2023-04-19 07:56:01,794 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 199c6b7f1f3a9353227e2a51c7965b15 in 59ms, sequenceid=242, compaction requested=false 2023-04-19 07:56:01,794 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:01,794 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,794 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:01,794 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:01,794 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,794 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:01,794 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:01,797 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,797 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:01,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b8fc3f4[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:01,799 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:01,799 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,800 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,800 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:01,802 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:01,802 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:01,803 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:01,803 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:01,803 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,803 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:01,807 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:01,807 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:01,808 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:01,808 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:01,808 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:01,809 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,809 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:01,809 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,811 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,816 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:01,816 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:01,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:01,816 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:01,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:01,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:01,817 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:01,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:01,817 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:01,817 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:01,817 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:01,817 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,817 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,817 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,817 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,817 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,917 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:01,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:01,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:01,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:01,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:01,920 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:01,920 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:01,920 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:01,920 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:01,921 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,921 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,922 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:01,922 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:01,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:01,923 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,923 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:01,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:01,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:01,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:01,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:01,924 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,924 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:01,924 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:01,924 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:01,925 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,925 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,926 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,926 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:01,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ed62245[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:01,927 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:01,927 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,928 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:01,928 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:01,928 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:01,929 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,929 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:01,931 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:01,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:01,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:01,935 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:01,935 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:01,935 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:01,935 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:01,935 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:01,936 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,936 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,936 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:01,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:01,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:01,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:01,937 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,939 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:01,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:01,942 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:01,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:01,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:01,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:01,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:01,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:01,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:01,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:01,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:01,942 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:01,942 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:01,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:02,005 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:56:02,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:02,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:02,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:02,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:02,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:02,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:02,043 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:02,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:02,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:02,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:02,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:02,046 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:02,046 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,046 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:02,046 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:02,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,048 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,048 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,048 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,048 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:02,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:02,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:02,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:02,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:02,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:02,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:02,050 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,050 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:02,050 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:02,050 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:02,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:02,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:02,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:02,058 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:02,060 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:02,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6360d09b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:02,060 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,061 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,061 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:02,061 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:02,061 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:02,061 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,061 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:02,062 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:02,062 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,062 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:02,062 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:02,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:02,065 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:02,066 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:02,066 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:02,066 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:02,066 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:02,070 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,070 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,070 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,071 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,072 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,074 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,074 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:02,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:02,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:02,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,075 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:02,075 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:02,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,082 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/68ad19c82a64478c83878319e145d17f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/68ad19c82a64478c83878319e145d17f 2023-04-19 07:56:02,086 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/A of 199c6b7f1f3a9353227e2a51c7965b15 into 68ad19c82a64478c83878319e145d17f(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:02,086 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:02,086 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/A, priority=13, startTime=1681890961649; duration=0sec 2023-04-19 07:56:02,086 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:02,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681890962066, max timestamp=1681890962100Num uniques 859; 2023-04-19 07:56:02,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 07:56:02,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681890962066, max timestamp=1681890962100Num uniques 859; 2023-04-19 07:56:02,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681890962066, max timestamp=1681890962100Num uniques 859; 2023-04-19 07:56:02,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 07:56:02,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-19 07:56:02,109 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/3dd76cce922945188e8b037f3800447f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/3dd76cce922945188e8b037f3800447f 2023-04-19 07:56:02,114 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/C of 199c6b7f1f3a9353227e2a51c7965b15 into 3dd76cce922945188e8b037f3800447f(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:02,114 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:02,114 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/C, priority=13, startTime=1681890961649; duration=0sec 2023-04-19 07:56:02,114 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:02,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891022131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891022131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891022132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891022133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891022133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,175 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:02,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:02,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:02,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:02,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:02,179 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:02,179 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,179 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:02,179 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:02,179 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,179 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,180 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,181 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,181 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:02,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:02,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:02,182 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:02,182 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,182 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:02,182 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:02,182 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:02,182 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,182 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:02,182 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:02,182 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,182 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:02,182 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:02,183 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,183 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:02,183 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:02,184 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,184 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,184 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,184 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:02,186 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5351aaff[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:02,186 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,188 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:02,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:02,190 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:02,190 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,190 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:02,191 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:02,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:02,204 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:02,204 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:02,204 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:02,204 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:02,204 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:02,206 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,206 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,206 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,206 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,208 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,210 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,211 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,211 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,211 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,211 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,211 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:02,211 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,211 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,211 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,211 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:02,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:02,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,211 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:02,211 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:02,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891022234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891022240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891022243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891022245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891022245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,311 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:02,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:02,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:02,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:02,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:02,315 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:02,315 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,315 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:02,315 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:02,315 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,315 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,317 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,317 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,318 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,318 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:02,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:02,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:02,319 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,319 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:02,319 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:02,319 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:02,319 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,320 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:02,320 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:02,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,321 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,321 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,321 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:02,322 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:02,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66744302[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:02,322 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,323 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:02,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:02,323 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:02,323 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,324 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:02,325 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,325 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:02,325 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,325 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:02,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:02,327 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:02,327 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:02,327 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:02,327 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:02,327 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:02,330 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,330 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,330 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,330 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,331 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,335 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,335 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,335 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,335 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,335 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,335 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,335 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,335 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,335 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,335 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:02,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:02,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:02,335 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:02,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,335 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:02,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,413 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/aa436fda9d074f08af62de80f17fb6df 2023-04-19 07:56:02,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/93d6f1d5d857462da99b3e75c2e933d0 2023-04-19 07:56:02,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/bb770ec1981e403289fa9cbd0c7b51a4 2023-04-19 07:56:02,436 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:02,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:02,437 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/aa436fda9d074f08af62de80f17fb6df as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/aa436fda9d074f08af62de80f17fb6df 2023-04-19 07:56:02,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:02,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:02,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:02,440 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:02,440 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,440 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:02,440 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:02,440 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,440 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,441 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,441 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,442 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/aa436fda9d074f08af62de80f17fb6df, entries=150, sequenceid=253, filesize=11.8 K 2023-04-19 07:56:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,442 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,442 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/93d6f1d5d857462da99b3e75c2e933d0 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/93d6f1d5d857462da99b3e75c2e933d0 2023-04-19 07:56:02,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:02,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:02,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:02,444 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:02,444 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,445 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:02,445 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:02,445 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,445 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:02,445 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:02,445 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,445 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:02,445 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:02,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891022441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/93d6f1d5d857462da99b3e75c2e933d0, entries=100, sequenceid=253, filesize=9.4 K 2023-04-19 07:56:02,447 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,447 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,447 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,447 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:02,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/bb770ec1981e403289fa9cbd0c7b51a4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/bb770ec1981e403289fa9cbd0c7b51a4 2023-04-19 07:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:02,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19cf8183[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:02,450 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:02,450 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,451 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/bb770ec1981e403289fa9cbd0c7b51a4, entries=100, sequenceid=253, filesize=9.4 K 2023-04-19 07:56:02,452 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 199c6b7f1f3a9353227e2a51c7965b15 in 447ms, sequenceid=253, compaction requested=true 2023-04-19 07:56:02,452 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,452 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:02,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:02,452 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:02,452 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:02,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:02,452 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:02,452 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,452 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:02,452 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:02,452 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:02,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:02,453 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:02,453 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/B is initiating minor compaction (all files) 2023-04-19 07:56:02,453 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/B in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,453 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/69357012cc4c424aaa6cec4332778d58, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/3a47a84c53d94ded89fab4480be777d8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/93d6f1d5d857462da99b3e75c2e933d0] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=33.2 K 2023-04-19 07:56:02,454 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:02,454 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36344 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:02,454 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,454 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:02,454 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 69357012cc4c424aaa6cec4332778d58, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681890959937 2023-04-19 07:56:02,454 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/A is initiating minor compaction (all files) 2023-04-19 07:56:02,454 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/A in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,454 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/68ad19c82a64478c83878319e145d17f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/914b73a88b1f436eacd250184ab831af, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/aa436fda9d074f08af62de80f17fb6df] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=35.5 K 2023-04-19 07:56:02,454 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 3a47a84c53d94ded89fab4480be777d8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681890960808 2023-04-19 07:56:02,454 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,455 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 93d6f1d5d857462da99b3e75c2e933d0, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681890961977 2023-04-19 07:56:02,455 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 68ad19c82a64478c83878319e145d17f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681890959937 2023-04-19 07:56:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,455 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 914b73a88b1f436eacd250184ab831af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681890960808 2023-04-19 07:56:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,456 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting aa436fda9d074f08af62de80f17fb6df, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681890961977 2023-04-19 07:56:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:02,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:02,458 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:02,458 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:02,458 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:02,458 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:02,458 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:02,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:02,459 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-19 07:56:02,459 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:02,459 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:02,459 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:02,459 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:02,459 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:02,459 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:02,460 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,460 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,460 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,461 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,462 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,468 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,468 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,468 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,468 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,468 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,468 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,468 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:02,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:02,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:02,468 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,469 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#B#compaction#415 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:02,470 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:02,470 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:02,471 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,471 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,474 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#A#compaction#416 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:02,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891022465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891022470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891022473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891022474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,488 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/811ccea87c6b4d82863c359016df2857 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/811ccea87c6b4d82863c359016df2857 2023-04-19 07:56:02,492 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/B of 199c6b7f1f3a9353227e2a51c7965b15 into 811ccea87c6b4d82863c359016df2857(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:02,492 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:02,492 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/B, priority=13, startTime=1681890962452; duration=0sec 2023-04-19 07:56:02,492 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:02,492 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:02,493 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:02,493 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/C is initiating minor compaction (all files) 2023-04-19 07:56:02,493 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/C in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,493 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/3dd76cce922945188e8b037f3800447f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b10e1361dd9b4b36a269c6079ff9affa, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/bb770ec1981e403289fa9cbd0c7b51a4] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=33.2 K 2023-04-19 07:56:02,494 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 3dd76cce922945188e8b037f3800447f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1681890959937 2023-04-19 07:56:02,494 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting b10e1361dd9b4b36a269c6079ff9affa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681890960808 2023-04-19 07:56:02,494 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting bb770ec1981e403289fa9cbd0c7b51a4, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681890961977 2023-04-19 07:56:02,500 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#C#compaction#417 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:02,508 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/8228f57684dc4dc3bd048b89447cb159 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/8228f57684dc4dc3bd048b89447cb159 2023-04-19 07:56:02,511 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/C of 199c6b7f1f3a9353227e2a51c7965b15 into 8228f57684dc4dc3bd048b89447cb159(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:02,512 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:02,512 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/C, priority=13, startTime=1681890962452; duration=0sec 2023-04-19 07:56:02,512 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:02,570 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:02,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:02,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:02,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:02,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:02,574 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:02,574 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,574 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:02,574 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:02,574 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,575 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,576 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,576 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:02,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:02,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:02,578 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,578 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:02,578 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:02,578 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:02,579 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,579 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:02,580 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,580 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,580 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,580 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891022576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891022577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:02,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cce6e29[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:02,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:02,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,585 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:02,586 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:02,586 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:02,586 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,586 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:02,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891022583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891022583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,591 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,591 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:02,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:02,594 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:02,594 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:02,594 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:02,594 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:02,594 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:02,598 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,598 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,598 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,599 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,600 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:02,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:02,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,602 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,603 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,603 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:02,603 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,603 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:02,603 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:02,603 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,703 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:02,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:02,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:02,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:02,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:02,707 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:02,707 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,707 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:02,707 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:02,707 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,707 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,708 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,708 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,709 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,709 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:02,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:02,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:02,710 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,710 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:02,710 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:02,710 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:02,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:02,712 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,713 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,713 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:02,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2296251d[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:02,714 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:02,714 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,715 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,716 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:02,716 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:02,716 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:02,716 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,716 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:02,717 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,717 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:02,717 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,717 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:02,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:02,723 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:02,723 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:02,723 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:02,723 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:02,723 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:02,724 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,725 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,725 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,725 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,727 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,730 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,730 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:02,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:02,730 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,730 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,730 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,730 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,730 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:02,730 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:02,730 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,730 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,730 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,730 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891022746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891022782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891022783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891022790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:02,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891022790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,830 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:02,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:02,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:02,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:02,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:02,834 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:02,834 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,834 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:02,834 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:02,834 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,834 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,835 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,835 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,836 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,836 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:02,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:02,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:02,837 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,837 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:02,837 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:02,837 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:02,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,838 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:02,838 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:02,839 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,839 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,839 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,839 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:02,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@663e0e1e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:02,840 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:02,841 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,842 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:02,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:02,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:02,842 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,842 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:02,844 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:02,844 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,844 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,844 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:02,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:02,847 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:02,847 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:02,847 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:02,847 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:02,847 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:02,848 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,848 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,848 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,849 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,850 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,852 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:02,852 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,853 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,853 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,853 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,853 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:02,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:02,852 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,853 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,853 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,853 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:02,853 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,853 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:02,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,884 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/d4a9ee4f3d42472cbfbb79a9ee9d209a 2023-04-19 07:56:02,889 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/0f73bfb47cbf4171ba23dbb3e16f9a43 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/0f73bfb47cbf4171ba23dbb3e16f9a43 2023-04-19 07:56:02,894 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/A of 199c6b7f1f3a9353227e2a51c7965b15 into 0f73bfb47cbf4171ba23dbb3e16f9a43(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:02,894 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:02,894 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/A, priority=13, startTime=1681890962452; duration=0sec 2023-04-19 07:56:02,894 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:02,953 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:02,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:02,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:02,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:02,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:02,957 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:02,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:02,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:02,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,962 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,962 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,963 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,963 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:02,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:02,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:02,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:02,964 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:02,964 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:02,964 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:02,964 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:02,964 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:02,964 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,964 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:02,965 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:02,965 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:02,965 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:02,965 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:02,965 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,965 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:02,965 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:02,966 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,966 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,966 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,967 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:02,968 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:02,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39858c3[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:02,968 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,969 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,969 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:02,969 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:02,969 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:02,971 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,971 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:02,971 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,971 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:02,971 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:02,971 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:02,971 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:02,971 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:02,971 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:02,971 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:02,971 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:02,973 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,973 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:02,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,973 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:02,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:02,974 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:02,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,975 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:02,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:02,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:02,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:02,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:02,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:02,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:02,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:02,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:02,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:02,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:02,979 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:02,979 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:02,979 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:02,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:03,079 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:03,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:03,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:03,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:03,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:03,083 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:03,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:03,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:03,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:03,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,084 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:03,084 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:03,085 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,085 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:03,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:03,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:03,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:03,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:03,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:03,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:03,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:03,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:03,086 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:03,086 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:03,086 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:03,086 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:03,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:03,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:03,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:03,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:03,088 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:03,089 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:03,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ca8c1bf[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:03,090 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,092 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,092 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:03,092 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:03,092 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:03,093 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,093 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:03,094 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,094 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,094 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:03,094 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:03,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891023092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:03,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891023092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:03,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891023092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:03,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891023093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:03,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:03,098 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:03,098 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:03,098 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:03,098 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:03,098 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:03,099 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,099 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,099 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,100 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,101 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,108 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,108 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:03,108 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:03,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:03,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:03,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:03,108 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,108 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:03,108 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:03,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:03,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,109 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,109 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:03,109 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:03,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:03,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:03,209 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:03,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:03,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:03,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:03,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:03,213 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:03,213 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:03,213 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:03,213 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:03,213 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,213 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,215 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:03,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:03,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:03,216 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,216 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:03,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:03,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:03,217 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:03,217 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:03,217 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:03,217 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:03,219 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:03,221 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:03,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f05a3fa[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:03,221 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,222 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,222 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:03,222 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:03,222 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:03,222 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,222 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:03,223 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,223 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:03,224 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:03,224 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:03,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:03,226 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:03,226 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:03,226 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:03,226 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:03,226 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:03,228 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,228 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,228 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:03,229 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,231 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,233 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,233 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:03,233 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,233 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:03,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:03,233 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:03,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:03,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:03,233 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,234 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,234 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,234 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,234 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,234 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:03,234 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:03,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:03,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:03,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:03,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:03,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37912 deadline: 1681891023251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,296 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/80b24338fc3c4c7d82c92152c17fea5f 2023-04-19 07:56:03,334 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:03,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:03,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:03,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:03,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:03,337 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:03,337 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:03,337 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:03,337 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:03,337 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,337 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,338 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:03,338 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:03,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:03,339 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,339 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:03,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:03,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:03,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:03,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:03,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:03,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:03,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:03,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:03,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:03,340 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:03,340 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:03,340 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:03,340 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:03,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:03,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:03,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:03,341 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:03,345 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,345 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,345 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,345 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:03,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:03,347 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9dd00fa[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:03,347 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:03,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:03,349 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:03,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:03,352 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:03,352 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:03,352 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:03,352 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:03,353 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:03,354 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,354 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,354 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:03,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:03,354 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,356 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,359 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:03,359 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:03,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:03,359 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:03,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:03,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:03,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:03,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:03,359 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:03,359 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:03,359 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,360 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,360 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,360 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,360 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,360 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,360 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,460 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:03,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:03,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:03,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:03,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:03,463 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:03,463 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:03,463 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:03,463 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:03,464 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,464 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,465 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:03,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:03,465 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,466 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,466 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:03,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:03,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:03,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:03,467 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:03,467 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:03,467 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:03,467 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:03,467 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:03,467 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:03,467 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:03,467 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:03,467 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:03,467 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:03,467 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:03,467 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,467 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:03,467 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:03,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:03,471 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:03,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48421f42[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:03,471 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,472 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,472 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,472 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:03,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:03,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:03,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:03,474 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:03,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:03,477 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:03,477 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:03,477 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:03,477 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:03,477 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:03,478 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,478 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:03,478 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:03,479 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,480 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,484 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:03,484 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:03,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:03,484 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:03,484 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,484 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,484 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,485 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,485 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,484 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:03,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:03,485 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:03,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:03,485 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:03,485 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:03,585 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:03,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:03,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:03,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:03,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:03,589 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:03,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:03,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:03,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:03,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,590 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:03,591 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:03,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:03,591 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,591 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:03,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:03,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:03,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:03,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:03,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:03,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:03,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:03,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:03,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:03,593 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:03,593 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:03,593 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:03,593 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:03,593 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:03,593 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:03,593 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,593 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:03,593 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:03,595 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,595 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,595 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,595 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:03,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:03,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22d9acc[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:03,597 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:03,597 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,598 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,598 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:03,598 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:03,598 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:03,598 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,598 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:03,599 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:03,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:03,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:03,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891023597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:03,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37896 deadline: 1681891023598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:03,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891023598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:03,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:03,603 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:03,603 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:03,603 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:03,603 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:03,603 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:03,605 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,605 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,605 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,605 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,607 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:03,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891023601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,611 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:03,611 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:03,611 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,611 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,611 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,611 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:03,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:03,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:03,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:03,611 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,611 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:03,611 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,611 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:03,611 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:03,611 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:03,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:03,709 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/b035a25347304d00b23c7006648814f8 2023-04-19 07:56:03,711 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:03,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:03,713 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/d4a9ee4f3d42472cbfbb79a9ee9d209a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d4a9ee4f3d42472cbfbb79a9ee9d209a 2023-04-19 07:56:03,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:03,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:03,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:03,716 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:03,716 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:03,716 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:03,716 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:03,716 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,717 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,717 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d4a9ee4f3d42472cbfbb79a9ee9d209a, entries=200, sequenceid=291, filesize=14.3 K 2023-04-19 07:56:03,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:03,718 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,718 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/80b24338fc3c4c7d82c92152c17fea5f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/80b24338fc3c4c7d82c92152c17fea5f 2023-04-19 07:56:03,719 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,719 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:03,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:03,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:03,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:03,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:03,720 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:03,720 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:03,720 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:03,720 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:03,720 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:03,720 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:03,721 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:03,721 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:03,721 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:03,721 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:03,721 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:03,721 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,721 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:03,721 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:03,722 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/80b24338fc3c4c7d82c92152c17fea5f, entries=150, sequenceid=291, filesize=11.9 K 2023-04-19 07:56:03,722 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,722 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,723 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,723 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:03,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3085f300[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:03,724 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:03,724 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/b035a25347304d00b23c7006648814f8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b035a25347304d00b23c7006648814f8 2023-04-19 07:56:03,725 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,726 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:03,726 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:03,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:03,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:03,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b035a25347304d00b23c7006648814f8, entries=150, sequenceid=291, filesize=11.9 K 2023-04-19 07:56:03,729 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for 199c6b7f1f3a9353227e2a51c7965b15 in 1270ms, sequenceid=291, compaction requested=false 2023-04-19 07:56:03,729 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:03,732 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:03,732 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,732 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,732 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:03,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:03,736 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:03,736 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:03,736 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:03,736 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:03,736 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:03,738 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,738 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:03,738 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:03,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:03,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:03,738 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,740 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:03,742 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,742 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,742 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,742 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,742 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,742 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:03,742 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:03,742 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,742 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:03,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:03,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:03,742 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:03,743 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:03,743 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:03,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:03,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:03,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:03,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:03,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:03,843 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:03,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:03,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:03,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:03,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:03,846 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:03,846 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:03,846 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:03,846 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:03,846 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,846 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,847 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:03,847 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:03,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:03,848 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:03,848 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:03,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:03,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:03,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:03,849 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:03,849 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:03,849 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:03,849 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:03,849 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:03,849 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:03,849 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:03,849 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 07:56:03,849 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:03,849 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:03,849 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:03,850 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:03,850 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:03,850 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:04,259 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/80b32d871a2f4ff7a08a1acc5e09bd1d 2023-04-19 07:56:04,260 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:04,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:04,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890964345, max timestamp=1681890964385Num uniques 859; 2023-04-19 07:56:04,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:04,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890964345, max timestamp=1681890964385Num uniques 859; 2023-04-19 07:56:04,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:04,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890964345, max timestamp=1681890964385Num uniques 859; 2023-04-19 07:56:04,388 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:04,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890964481, max timestamp=1681890964505Num uniques 1018; 2023-04-19 07:56:04,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:56:04,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890964481, max timestamp=1681890964505Num uniques 1018; 2023-04-19 07:56:04,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890964481, max timestamp=1681890964505Num uniques 1018; 2023-04-19 07:56:04,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:56:04,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:56:04,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:04,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891024610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:04,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681890964586, max timestamp=1681890964614Num uniques 1027; 2023-04-19 07:56:04,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-19 07:56:04,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681890964586, max timestamp=1681890964614Num uniques 1027; 2023-04-19 07:56:04,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681890964586, max timestamp=1681890964614Num uniques 1027; 2023-04-19 07:56:04,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-19 07:56:04,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-19 07:56:04,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:04,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891024639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:04,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:04,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891024642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:04,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:04,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891024643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:04,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:04,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891024648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:04,672 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/19c523d8f7994f73b28afe1574124e43 2023-04-19 07:56:04,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:04,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891024749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:04,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:04,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891024750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:04,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:04,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891024750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:04,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:04,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891024756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:04,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:04,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891024957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:04,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:04,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891024957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:04,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:04,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891024957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:04,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:04,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891024960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,082 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/a827c93349a5499a8d58d4994b2652dd 2023-04-19 07:56:05,085 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/80b32d871a2f4ff7a08a1acc5e09bd1d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/80b32d871a2f4ff7a08a1acc5e09bd1d 2023-04-19 07:56:05,089 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/80b32d871a2f4ff7a08a1acc5e09bd1d, entries=100, sequenceid=303, filesize=9.5 K 2023-04-19 07:56:05,090 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/19c523d8f7994f73b28afe1574124e43 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/19c523d8f7994f73b28afe1574124e43 2023-04-19 07:56:05,093 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/19c523d8f7994f73b28afe1574124e43, entries=100, sequenceid=303, filesize=9.5 K 2023-04-19 07:56:05,094 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/a827c93349a5499a8d58d4994b2652dd as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a827c93349a5499a8d58d4994b2652dd 2023-04-19 07:56:05,097 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a827c93349a5499a8d58d4994b2652dd, entries=100, sequenceid=303, filesize=9.5 K 2023-04-19 07:56:05,098 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 199c6b7f1f3a9353227e2a51c7965b15 in 1249ms, sequenceid=303, compaction requested=true 2023-04-19 07:56:05,098 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:05,098 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:05,098 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:05,099 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:05,099 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,099 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:05,099 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:05,101 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,101 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:05,102 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,102 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:05,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:05,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51beef3a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:05,104 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:05,104 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,105 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,105 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:05,105 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:05,105 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:05,106 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:05,106 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:05,106 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:05,106 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,107 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:05,107 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:05,107 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:05,107 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:05,107 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:05,107 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:05,107 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:05,108 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,108 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,108 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:05,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:05,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:05,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:05,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:05,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:05,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:05,110 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:05,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:05,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,111 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,114 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:05,114 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:05,114 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:05,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:05,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:05,114 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:05,114 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:05,114 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:05,114 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:05,114 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:05,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:05,114 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,115 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,115 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,115 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,214 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:05,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:05,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:05,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:05,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:05,222 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:05,222 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:05,222 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:05,222 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:05,222 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,223 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:05,224 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:05,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:05,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:05,224 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:05,224 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,224 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:05,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:05,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:05,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:05,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:05,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:05,226 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:05,226 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:05,226 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:05,226 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:05,227 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:05,227 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:05,227 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:05,227 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=140.89 KB heapSize=393.84 KB 2023-04-19 07:56:05,227 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:05,227 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:05,227 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:05,227 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:05,227 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:05,227 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:05,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:05,273 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:05,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:05,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891025275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:05,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891025275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:05,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891025282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:05,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891025283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:05,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891025384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:05,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891025385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:05,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891025388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:05,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891025391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:05,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37878 deadline: 1681891025591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:05,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37928 deadline: 1681891025592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:05,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891025592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:05,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891025599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,645 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/794d2e6c7b6f43c492fb2996f7869c66 2023-04-19 07:56:05,655 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/c3e589b42de443d39091468667f60938 2023-04-19 07:56:05,664 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/ac599d546e8342ec8d7c8fffa49d9a31 2023-04-19 07:56:05,669 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/794d2e6c7b6f43c492fb2996f7869c66 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/794d2e6c7b6f43c492fb2996f7869c66 2023-04-19 07:56:05,672 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/794d2e6c7b6f43c492fb2996f7869c66, entries=150, sequenceid=373, filesize=11.9 K 2023-04-19 07:56:05,673 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/c3e589b42de443d39091468667f60938 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/c3e589b42de443d39091468667f60938 2023-04-19 07:56:05,677 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/c3e589b42de443d39091468667f60938, entries=150, sequenceid=373, filesize=11.9 K 2023-04-19 07:56:05,677 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/ac599d546e8342ec8d7c8fffa49d9a31 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/ac599d546e8342ec8d7c8fffa49d9a31 2023-04-19 07:56:05,681 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/ac599d546e8342ec8d7c8fffa49d9a31, entries=150, sequenceid=373, filesize=11.9 K 2023-04-19 07:56:05,682 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~393.80 KB/403248, currentSize=40.25 KB/41220 for 199c6b7f1f3a9353227e2a51c7965b15 in 455ms, sequenceid=373, compaction requested=true 2023-04-19 07:56:05,682 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:05,682 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:05,682 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:05,682 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:05,682 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,682 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:05,682 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:05,684 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:05,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:05,686 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,686 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:05,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:05,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9de5240[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:05,688 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:05,688 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,690 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:05,690 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:05,690 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:05,690 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,690 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:05,692 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:05,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:05,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:05,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:05,695 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:05,695 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:05,695 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:05,696 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:05,696 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:05,697 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,697 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:05,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,697 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:05,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:05,698 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:05,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:05,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:05,701 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:05,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:05,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:05,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:05,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:05,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:05,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,703 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:05,704 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:05,704 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:05,704 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:05,704 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:05,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:05,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:05,804 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:05,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:05,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:05,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:05,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:05,807 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:05,807 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:05,807 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:05,807 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:05,807 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,808 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:05,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,809 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:05,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:05,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:05,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:05,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:05,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:05,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:05,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:05,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:05,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:05,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:05,810 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:05,811 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:05,811 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:05,811 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:05,811 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:05,811 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:05,811 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:05,811 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 07:56:05,811 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:05,811 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:05,811 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:05,811 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:05,811 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:05,811 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:05,820 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/d6357e0f085f4b1aa6035792391198bf 2023-04-19 07:56:05,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:05,900 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:05,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890965960, max timestamp=1681890965969Num uniques 859; 2023-04-19 07:56:05,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:05,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890965960, max timestamp=1681890965969Num uniques 859; 2023-04-19 07:56:05,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:05,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890965960, max timestamp=1681890965969Num uniques 859; 2023-04-19 07:56:05,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:06,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891026000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890965990, max timestamp=1681890966005Num uniques 1018; 2023-04-19 07:56:06,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890965990, max timestamp=1681890966005Num uniques 1018; 2023-04-19 07:56:06,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 07:56:06,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890965990, max timestamp=1681890966005Num uniques 1018; 2023-04-19 07:56:06,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 07:56:06,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-19 07:56:06,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891026042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891026050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891026051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891026109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891026153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891026160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891026160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,236 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/31569ca5d0ce4f8c8237546d7d27ad04 2023-04-19 07:56:06,248 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=382 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/ce14e03e2287471b976c21ad347e80c2 2023-04-19 07:56:06,253 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/d6357e0f085f4b1aa6035792391198bf as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d6357e0f085f4b1aa6035792391198bf 2023-04-19 07:56:06,257 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d6357e0f085f4b1aa6035792391198bf, entries=150, sequenceid=382, filesize=11.9 K 2023-04-19 07:56:06,258 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/31569ca5d0ce4f8c8237546d7d27ad04 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/31569ca5d0ce4f8c8237546d7d27ad04 2023-04-19 07:56:06,261 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/31569ca5d0ce4f8c8237546d7d27ad04, entries=150, sequenceid=382, filesize=11.9 K 2023-04-19 07:56:06,261 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/ce14e03e2287471b976c21ad347e80c2 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/ce14e03e2287471b976c21ad347e80c2 2023-04-19 07:56:06,265 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/ce14e03e2287471b976c21ad347e80c2, entries=150, sequenceid=382, filesize=11.9 K 2023-04-19 07:56:06,266 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 199c6b7f1f3a9353227e2a51c7965b15 in 455ms, sequenceid=382, compaction requested=true 2023-04-19 07:56:06,266 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:06,266 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:06,266 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:06,270 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:06,270 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:06,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:06,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:06,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:06,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@674b424d[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:06,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:06,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,273 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,273 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:06,273 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:06,273 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:06,273 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,273 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:06,274 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,274 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:06,274 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:06,275 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:06,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:06,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:06,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:06,279 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:06,279 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:06,279 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:06,279 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:06,279 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:06,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:06,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:06,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:06,285 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:06,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,286 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:06,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,288 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:06,288 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,288 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,288 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,288 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,288 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:06,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:06,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:06,288 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:06,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:06,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:06,288 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,288 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,288 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:06,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:06,288 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:06,288 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:06,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:06,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:06,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-04-19 07:56:06,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:06,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:06,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:06,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:06,320 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:06,320 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:06,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891026360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891026362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891026362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891026362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,389 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:06,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:06,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:06,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:06,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:06,392 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:06,392 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:06,392 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:06,392 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:06,392 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,392 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,393 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:06,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:06,394 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:06,395 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,395 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:06,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:06,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:06,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:06,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:06,396 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:06,397 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:06,397 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:06,397 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:06,397 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:06,397 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:06,399 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,399 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:06,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:06,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:06,400 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,400 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:06,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:06,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:06,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:06,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c9b057[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:06,401 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:06,401 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,403 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,403 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:06,403 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:06,403 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:06,403 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,403 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:06,404 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:06,405 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:06,405 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:06,405 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:06,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:06,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:06,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:06,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:06,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:06,407 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:06,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:06,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:06,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:06,407 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:06,408 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,408 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:06,409 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:06,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:06,409 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:06,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,410 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:06,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,414 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,414 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,414 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,414 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,414 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,414 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,414 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:06,414 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:06,414 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:06,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:06,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:06,414 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,414 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:06,414 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:06,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:06,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:06,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:06,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:06,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:06,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891026471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,515 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:06,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:06,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:06,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:06,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:06,518 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:06,518 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:06,518 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:06,518 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:06,519 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,519 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,526 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:06,526 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:06,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:06,526 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,526 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:06,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:06,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:06,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:06,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:06,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:06,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:06,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:06,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:06,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:06,528 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:06,528 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:06,528 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:06,528 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:06,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:06,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:06,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:06,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:06,530 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:06,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:06,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:06,530 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:06,531 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,531 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:06,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:06,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:06,532 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:06,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea8474b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:06,532 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,533 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,533 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:06,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,533 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:06,533 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:06,533 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:06,535 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:06,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:06,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:06,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:06,535 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:06,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:06,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:06,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:06,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:06,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:06,537 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:06,537 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:06,537 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:06,537 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:06,537 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:06,540 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,540 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,540 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:06,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:06,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:06,541 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,542 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:06,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,544 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:06,544 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:06,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:06,544 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,544 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:06,544 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:06,544 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,544 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,545 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,545 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,545 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:06,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:06,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:06,545 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:06,545 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:06,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:06,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891026627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,642 DEBUG [Thread-2959] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:56:06,645 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:06,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:06,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:06,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:06,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:06,648 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:06,649 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:06,649 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:06,649 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:06,649 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,649 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,650 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:06,650 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:06,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:06,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,651 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,651 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:06,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:06,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:06,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:06,652 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:06,652 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:06,652 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:06,652 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:06,652 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:06,652 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:06,652 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:06,653 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:06,653 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:06,653 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:06,653 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:06,653 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,653 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:06,653 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:06,654 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,654 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:06,655 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,655 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:06,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:06,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:06,656 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:06,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@259f428c[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:06,656 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,658 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,659 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:06,659 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:06,659 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:06,659 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,659 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:06,660 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,660 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:06,660 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:06,660 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:06,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:06,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:06,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:06,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:06,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:06,664 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:06,664 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:06,664 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:06,664 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:06,664 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:06,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:06,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:06,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:06,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:06,667 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:06,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,668 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:06,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:06,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,675 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:06,675 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:06,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:06,675 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:06,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:06,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:06,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:06,675 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:06,675 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,675 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,675 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,675 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:06,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:06,675 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:06,676 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:06,675 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:06,676 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891026670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891026671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891026672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:06,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891026686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/f090e36c516543df99081c5a9528157a 2023-04-19 07:56:06,745 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/91a84f4b709040e1886367a97f9216f6 2023-04-19 07:56:06,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/42750a62293f4e86a92ed1d79b0cbcae 2023-04-19 07:56:06,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/f090e36c516543df99081c5a9528157a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/f090e36c516543df99081c5a9528157a 2023-04-19 07:56:06,766 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/f090e36c516543df99081c5a9528157a, entries=200, sequenceid=436, filesize=14.3 K 2023-04-19 07:56:06,767 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/91a84f4b709040e1886367a97f9216f6 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/91a84f4b709040e1886367a97f9216f6 2023-04-19 07:56:06,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/91a84f4b709040e1886367a97f9216f6, entries=150, sequenceid=436, filesize=11.9 K 2023-04-19 07:56:06,771 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/42750a62293f4e86a92ed1d79b0cbcae as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/42750a62293f4e86a92ed1d79b0cbcae 2023-04-19 07:56:06,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/42750a62293f4e86a92ed1d79b0cbcae, entries=150, sequenceid=436, filesize=11.9 K 2023-04-19 07:56:06,775 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=33.54 KB/34350 for 199c6b7f1f3a9353227e2a51c7965b15 in 455ms, sequenceid=436, compaction requested=true 2023-04-19 07:56:06,775 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:06,775 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:06,775 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 07:56:06,775 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 07:56:06,775 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:06,775 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:06,776 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:06,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:06,778 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 75744 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 07:56:06,778 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/A is initiating minor compaction (all files) 2023-04-19 07:56:06,778 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/A in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:06,778 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/0f73bfb47cbf4171ba23dbb3e16f9a43, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d4a9ee4f3d42472cbfbb79a9ee9d209a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/80b32d871a2f4ff7a08a1acc5e09bd1d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/794d2e6c7b6f43c492fb2996f7869c66, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d6357e0f085f4b1aa6035792391198bf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/f090e36c516543df99081c5a9528157a] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=74.0 K 2023-04-19 07:56:06,778 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70864 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 07:56:06,778 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/B is initiating minor compaction (all files) 2023-04-19 07:56:06,778 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/B in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:06,779 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/811ccea87c6b4d82863c359016df2857, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/80b24338fc3c4c7d82c92152c17fea5f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/19c523d8f7994f73b28afe1574124e43, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/c3e589b42de443d39091468667f60938, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/31569ca5d0ce4f8c8237546d7d27ad04, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/91a84f4b709040e1886367a97f9216f6] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=69.2 K 2023-04-19 07:56:06,779 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 0f73bfb47cbf4171ba23dbb3e16f9a43, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681890960808 2023-04-19 07:56:06,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:06,779 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 811ccea87c6b4d82863c359016df2857, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681890960808 2023-04-19 07:56:06,779 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting d4a9ee4f3d42472cbfbb79a9ee9d209a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681890962130 2023-04-19 07:56:06,779 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 80b24338fc3c4c7d82c92152c17fea5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681890962130 2023-04-19 07:56:06,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:06,780 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 80b32d871a2f4ff7a08a1acc5e09bd1d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681890962469 2023-04-19 07:56:06,780 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 19c523d8f7994f73b28afe1574124e43, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681890962469 2023-04-19 07:56:06,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:06,780 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:06,780 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:06,780 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 794d2e6c7b6f43c492fb2996f7869c66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1681890964641 2023-04-19 07:56:06,780 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:06,780 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:06,781 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,781 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,782 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:06,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:06,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:06,782 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,782 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:06,782 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:06,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:06,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:06,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:06,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:06,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:06,783 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:06,783 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:06,783 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:06,783 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:06,783 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:06,783 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:06,784 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:06,784 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-19 07:56:06,784 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:06,784 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting c3e589b42de443d39091468667f60938, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1681890964641 2023-04-19 07:56:06,784 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:06,784 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting d6357e0f085f4b1aa6035792391198bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1681890965274 2023-04-19 07:56:06,784 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:06,784 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:06,784 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:06,784 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 31569ca5d0ce4f8c8237546d7d27ad04, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1681890965274 2023-04-19 07:56:06,785 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:06,785 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting f090e36c516543df99081c5a9528157a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681890966016 2023-04-19 07:56:06,785 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 91a84f4b709040e1886367a97f9216f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681890966049 2023-04-19 07:56:06,802 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#B#compaction#433 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:06,805 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#A#compaction#434 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:06,825 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/f51d8afd3f234cebad4fdb4a4f3672ab as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/f51d8afd3f234cebad4fdb4a4f3672ab 2023-04-19 07:56:06,829 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/B of 199c6b7f1f3a9353227e2a51c7965b15 into f51d8afd3f234cebad4fdb4a4f3672ab(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:06,829 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:06,830 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/B, priority=10, startTime=1681890966775; duration=0sec 2023-04-19 07:56:06,830 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:06,830 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 07:56:06,831 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70864 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 07:56:06,831 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/C is initiating minor compaction (all files) 2023-04-19 07:56:06,831 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/C in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:06,831 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/8228f57684dc4dc3bd048b89447cb159, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b035a25347304d00b23c7006648814f8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a827c93349a5499a8d58d4994b2652dd, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/ac599d546e8342ec8d7c8fffa49d9a31, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/ce14e03e2287471b976c21ad347e80c2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/42750a62293f4e86a92ed1d79b0cbcae] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=69.2 K 2023-04-19 07:56:06,831 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 8228f57684dc4dc3bd048b89447cb159, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681890960808 2023-04-19 07:56:06,832 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting b035a25347304d00b23c7006648814f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1681890962130 2023-04-19 07:56:06,832 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting a827c93349a5499a8d58d4994b2652dd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681890962469 2023-04-19 07:56:06,832 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting ac599d546e8342ec8d7c8fffa49d9a31, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1681890964641 2023-04-19 07:56:06,832 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting ce14e03e2287471b976c21ad347e80c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=382, earliestPutTs=1681890965274 2023-04-19 07:56:06,833 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 42750a62293f4e86a92ed1d79b0cbcae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681890966049 2023-04-19 07:56:06,842 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#C#compaction#435 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:06,856 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/9e7f4ac311544aaa9998c7f96a86e9b5 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/9e7f4ac311544aaa9998c7f96a86e9b5 2023-04-19 07:56:06,859 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/C of 199c6b7f1f3a9353227e2a51c7965b15 into 9e7f4ac311544aaa9998c7f96a86e9b5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:06,859 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:06,859 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/C, priority=10, startTime=1681890966775; duration=0sec 2023-04-19 07:56:06,860 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:06,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:06,996 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:07,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890967040, max timestamp=1681890967072Num uniques 859; 2023-04-19 07:56:07,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890967040, max timestamp=1681890967072Num uniques 859; 2023-04-19 07:56:07,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:07,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890967040, max timestamp=1681890967072Num uniques 859; 2023-04-19 07:56:07,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:07,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:07,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890967134, max timestamp=1681890967155Num uniques 1018; 2023-04-19 07:56:07,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:56:07,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890967134, max timestamp=1681890967155Num uniques 1018; 2023-04-19 07:56:07,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890967134, max timestamp=1681890967155Num uniques 1018; 2023-04-19 07:56:07,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:56:07,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:56:07,210 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/5d7bbd32126940e99c2ed1f2e465244b 2023-04-19 07:56:07,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890967199, max timestamp=1681890967211Num uniques 1027; 2023-04-19 07:56:07,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-19 07:56:07,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890967199, max timestamp=1681890967211Num uniques 1027; 2023-04-19 07:56:07,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-19 07:56:07,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681890967199, max timestamp=1681890967211Num uniques 1027; 2023-04-19 07:56:07,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-19 07:56:07,235 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/7fdac7e3e3eb4ad59df8eb5a795c0721 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/7fdac7e3e3eb4ad59df8eb5a795c0721 2023-04-19 07:56:07,242 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/A of 199c6b7f1f3a9353227e2a51c7965b15 into 7fdac7e3e3eb4ad59df8eb5a795c0721(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:07,243 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:07,243 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/A, priority=10, startTime=1681890966775; duration=0sec 2023-04-19 07:56:07,243 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:07,246 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/1ea82af70386489188115e8a142465c5 2023-04-19 07:56:07,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681890967249, max timestamp=1681890967269Num uniques 1086; 2023-04-19 07:56:07,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-19 07:56:07,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681890967249, max timestamp=1681890967269Num uniques 1086; 2023-04-19 07:56:07,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-19 07:56:07,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681890967249, max timestamp=1681890967269Num uniques 1086; 2023-04-19 07:56:07,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=15386, suffix heap size=140912,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=336 2023-04-19 07:56:07,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891027306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891027307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891027312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891027313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891027416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891027419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891027419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891027420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891027623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891027625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891027626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891027626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,668 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/a1f6d12bd18348bb8f5bdfd045062702 2023-04-19 07:56:07,672 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/5d7bbd32126940e99c2ed1f2e465244b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/5d7bbd32126940e99c2ed1f2e465244b 2023-04-19 07:56:07,675 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/5d7bbd32126940e99c2ed1f2e465244b, entries=100, sequenceid=444, filesize=9.5 K 2023-04-19 07:56:07,676 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/1ea82af70386489188115e8a142465c5 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/1ea82af70386489188115e8a142465c5 2023-04-19 07:56:07,679 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/1ea82af70386489188115e8a142465c5, entries=100, sequenceid=444, filesize=9.5 K 2023-04-19 07:56:07,680 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/a1f6d12bd18348bb8f5bdfd045062702 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a1f6d12bd18348bb8f5bdfd045062702 2023-04-19 07:56:07,691 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a1f6d12bd18348bb8f5bdfd045062702, entries=100, sequenceid=444, filesize=9.5 K 2023-04-19 07:56:07,692 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 199c6b7f1f3a9353227e2a51c7965b15 in 908ms, sequenceid=444, compaction requested=false 2023-04-19 07:56:07,692 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:07,692 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:07,692 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:07,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:07,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:07,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:07,694 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,695 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:07,695 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:07,695 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:07,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:07,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:07,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@187b2558[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:07,696 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:07,697 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:07,698 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:07,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:07,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:07,698 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:07,698 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:07,698 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:07,698 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,698 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:07,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:07,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:07,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:07,701 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:07,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:07,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:07,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:07,703 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:07,703 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:07,703 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:07,703 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:07,703 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:07,704 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:07,704 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:07,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:07,704 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:07,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:07,705 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:07,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:07,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:07,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:07,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:07,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,709 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,709 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:07,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:07,709 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:07,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:07,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:07,709 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:07,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:07,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:07,709 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:07,709 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:07,709 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:07,709 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:07,709 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,709 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:07,709 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:07,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:07,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:07,709 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:07,809 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:07,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:07,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:07,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:07,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:07,813 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:07,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:07,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:07,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:07,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:07,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:07,814 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,814 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:07,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:07,815 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,815 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:07,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:07,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:07,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:07,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:07,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:07,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:07,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:07,816 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:07,816 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:07,817 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=154.31 KB heapSize=428.16 KB 2023-04-19 07:56:07,817 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:07,817 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:07,817 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:07,817 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:07,817 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:07,817 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:07,826 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/8f59454e67e941ddaa4193d2ab33f069 2023-04-19 07:56:07,840 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/160a901bcb864794899e3278d7abbbd3 2023-04-19 07:56:07,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:07,934 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:07,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891027935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891027936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891027937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:07,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:07,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891027938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891028040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891028042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891028046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891028046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,252 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/f6ba06c380a4426fa5733906aeaa746d 2023-04-19 07:56:08,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891028246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891028246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,256 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/8f59454e67e941ddaa4193d2ab33f069 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/8f59454e67e941ddaa4193d2ab33f069 2023-04-19 07:56:08,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891028252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891028252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,268 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/8f59454e67e941ddaa4193d2ab33f069, entries=150, sequenceid=535, filesize=11.9 K 2023-04-19 07:56:08,269 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/160a901bcb864794899e3278d7abbbd3 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/160a901bcb864794899e3278d7abbbd3 2023-04-19 07:56:08,272 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/160a901bcb864794899e3278d7abbbd3, entries=150, sequenceid=535, filesize=11.9 K 2023-04-19 07:56:08,272 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/f6ba06c380a4426fa5733906aeaa746d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/f6ba06c380a4426fa5733906aeaa746d 2023-04-19 07:56:08,276 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/f6ba06c380a4426fa5733906aeaa746d, entries=150, sequenceid=535, filesize=11.9 K 2023-04-19 07:56:08,277 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~428.11 KB/438384, currentSize=33.54 KB/34350 for 199c6b7f1f3a9353227e2a51c7965b15 in 460ms, sequenceid=535, compaction requested=true 2023-04-19 07:56:08,277 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:08,277 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:08,277 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:08,277 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:08,277 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,277 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:08,277 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:08,279 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,279 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,280 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,280 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:08,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d056508[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:08,283 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:08,283 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:08,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:08,284 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:08,285 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,285 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:08,285 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,286 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:08,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:08,289 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:08,289 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:08,289 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:08,289 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:08,289 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:08,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,291 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:08,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,292 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,292 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,295 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,301 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,301 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,301 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:08,301 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:08,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:08,301 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,301 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:08,301 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:08,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:08,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:08,301 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,301 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,301 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,302 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:08,302 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:08,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:08,402 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:08,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:08,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:08,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:08,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:08,405 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:08,405 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:08,405 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:08,405 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:08,406 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,406 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,407 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:08,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:08,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:08,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:08,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:08,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:08,409 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:08,409 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:08,409 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:08,409 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:08,409 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:08,409 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:08,409 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:08,409 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-19 07:56:08,409 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:08,410 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:08,410 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:08,410 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:08,410 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:08,410 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:08,423 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/20a0fd4d32584c8984ce138d18b85dc9 2023-04-19 07:56:08,438 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/b987c1c8316f4a4b8beeae4fb0ab8209 2023-04-19 07:56:08,454 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/6ead41732a8249ef9767e47d083026b4 2023-04-19 07:56:08,458 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/20a0fd4d32584c8984ce138d18b85dc9 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/20a0fd4d32584c8984ce138d18b85dc9 2023-04-19 07:56:08,462 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/20a0fd4d32584c8984ce138d18b85dc9, entries=100, sequenceid=543, filesize=9.5 K 2023-04-19 07:56:08,463 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/b987c1c8316f4a4b8beeae4fb0ab8209 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/b987c1c8316f4a4b8beeae4fb0ab8209 2023-04-19 07:56:08,468 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/b987c1c8316f4a4b8beeae4fb0ab8209, entries=100, sequenceid=543, filesize=9.5 K 2023-04-19 07:56:08,469 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/6ead41732a8249ef9767e47d083026b4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/6ead41732a8249ef9767e47d083026b4 2023-04-19 07:56:08,473 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/6ead41732a8249ef9767e47d083026b4, entries=100, sequenceid=543, filesize=9.5 K 2023-04-19 07:56:08,474 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 199c6b7f1f3a9353227e2a51c7965b15 in 65ms, sequenceid=543, compaction requested=true 2023-04-19 07:56:08,474 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:08,474 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:08,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:08,475 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,475 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,476 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,476 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:08,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@310bedfc[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:08,477 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:08,477 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,478 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,478 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:08,479 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:08,479 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:08,479 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,479 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:08,481 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,481 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:08,481 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,481 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:08,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:08,484 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:08,484 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:08,484 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:08,484 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:08,484 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:08,486 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,486 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,486 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:08,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:08,486 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,489 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,492 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,492 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,492 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,492 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:08,492 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:08,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:08,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:08,492 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,493 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,493 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:08,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:08,493 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:08,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:08,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:08,493 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,493 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:08,493 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:08,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:08,565 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:56:08,565 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:08,565 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:08,565 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:08,565 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:08,565 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:08,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:08,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/3c411b78a9b74d75b251ebd85714d54d 2023-04-19 07:56:08,582 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/caf743df133e4dd2bc56b7cb9d6710b5 2023-04-19 07:56:08,590 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/5b202ca9b6d14e0a81b29f007e7db001 2023-04-19 07:56:08,593 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/3c411b78a9b74d75b251ebd85714d54d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/3c411b78a9b74d75b251ebd85714d54d 2023-04-19 07:56:08,593 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:08,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:08,598 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/3c411b78a9b74d75b251ebd85714d54d, entries=100, sequenceid=553, filesize=9.5 K 2023-04-19 07:56:08,599 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/caf743df133e4dd2bc56b7cb9d6710b5 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/caf743df133e4dd2bc56b7cb9d6710b5 2023-04-19 07:56:08,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:08,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:08,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:08,601 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:08,601 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:08,601 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:08,601 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:08,602 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,602 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,603 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/caf743df133e4dd2bc56b7cb9d6710b5, entries=100, sequenceid=553, filesize=9.5 K 2023-04-19 07:56:08,604 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/5b202ca9b6d14e0a81b29f007e7db001 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/5b202ca9b6d14e0a81b29f007e7db001 2023-04-19 07:56:08,605 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,606 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:08,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:08,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:08,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890968590, max timestamp=1681890968605Num uniques 859; 2023-04-19 07:56:08,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:56:08,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890968590, max timestamp=1681890968605Num uniques 859; 2023-04-19 07:56:08,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:56:08,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890968590, max timestamp=1681890968605Num uniques 859; 2023-04-19 07:56:08,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:56:08,608 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,608 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:08,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:08,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:08,609 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/5b202ca9b6d14e0a81b29f007e7db001, entries=100, sequenceid=553, filesize=9.5 K 2023-04-19 07:56:08,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:08,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:08,610 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:08,610 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:08,610 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:08,610 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=40.25 KB/41220 for 199c6b7f1f3a9353227e2a51c7965b15 in 45ms, sequenceid=553, compaction requested=true 2023-04-19 07:56:08,610 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:08,610 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:08,610 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:08,610 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:08,610 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:08,610 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:56:08,610 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=40.25 KB heapSize=101.39 KB 2023-04-19 07:56:08,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:08,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:08,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 07:56:08,610 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:08,610 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:56:08,611 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:08,611 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:08,611 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:08,611 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:08,611 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:08,612 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 54144 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:56:08,612 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/A is initiating minor compaction (all files) 2023-04-19 07:56:08,612 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/A in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:08,612 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/7fdac7e3e3eb4ad59df8eb5a795c0721, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/5d7bbd32126940e99c2ed1f2e465244b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/8f59454e67e941ddaa4193d2ab33f069, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/20a0fd4d32584c8984ce138d18b85dc9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/3c411b78a9b74d75b251ebd85714d54d] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=52.9 K 2023-04-19 07:56:08,612 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 54144 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:56:08,612 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/B is initiating minor compaction (all files) 2023-04-19 07:56:08,612 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/B in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:08,612 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 7fdac7e3e3eb4ad59df8eb5a795c0721, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681890966049 2023-04-19 07:56:08,612 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/f51d8afd3f234cebad4fdb4a4f3672ab, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/1ea82af70386489188115e8a142465c5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/160a901bcb864794899e3278d7abbbd3, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/b987c1c8316f4a4b8beeae4fb0ab8209, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/caf743df133e4dd2bc56b7cb9d6710b5] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=52.9 K 2023-04-19 07:56:08,613 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 5d7bbd32126940e99c2ed1f2e465244b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681890966353 2023-04-19 07:56:08,613 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting f51d8afd3f234cebad4fdb4a4f3672ab, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681890966049 2023-04-19 07:56:08,613 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 1ea82af70386489188115e8a142465c5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681890966353 2023-04-19 07:56:08,613 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 8f59454e67e941ddaa4193d2ab33f069, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681890967304 2023-04-19 07:56:08,614 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 160a901bcb864794899e3278d7abbbd3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681890967304 2023-04-19 07:56:08,614 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 20a0fd4d32584c8984ce138d18b85dc9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1681890967936 2023-04-19 07:56:08,614 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting b987c1c8316f4a4b8beeae4fb0ab8209, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1681890967936 2023-04-19 07:56:08,614 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 3c411b78a9b74d75b251ebd85714d54d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1681890968563 2023-04-19 07:56:08,614 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting caf743df133e4dd2bc56b7cb9d6710b5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1681890968563 2023-04-19 07:56:08,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:08,615 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:08,619 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/b535673690244e0ea0ee065441be78b7 2023-04-19 07:56:08,630 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#B#compaction#448 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:08,631 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#A#compaction#450 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:08,652 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/46ff68515b7f4c0d8075db6600e61e5b 2023-04-19 07:56:08,662 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/2d3b1696ef6a48fb98076c6c0210a27d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2d3b1696ef6a48fb98076c6c0210a27d 2023-04-19 07:56:08,666 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/71ef49dea68448e2b7301533e5fd3079 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/71ef49dea68448e2b7301533e5fd3079 2023-04-19 07:56:08,671 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/d4f911fb532b4fe19213914f1bc5ea38 2023-04-19 07:56:08,674 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/b535673690244e0ea0ee065441be78b7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/b535673690244e0ea0ee065441be78b7 2023-04-19 07:56:08,677 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/B of 199c6b7f1f3a9353227e2a51c7965b15 into 2d3b1696ef6a48fb98076c6c0210a27d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:08,677 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/A of 199c6b7f1f3a9353227e2a51c7965b15 into 71ef49dea68448e2b7301533e5fd3079(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:08,677 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:08,677 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:08,677 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/A, priority=11, startTime=1681890968610; duration=0sec 2023-04-19 07:56:08,677 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/B, priority=11, startTime=1681890968610; duration=0sec 2023-04-19 07:56:08,677 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:08,677 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:08,677 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:56:08,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890968646, max timestamp=1681890968664Num uniques 859; 2023-04-19 07:56:08,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:08,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890968646, max timestamp=1681890968664Num uniques 859; 2023-04-19 07:56:08,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890968646, max timestamp=1681890968664Num uniques 859; 2023-04-19 07:56:08,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:08,679 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:08,679 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/b535673690244e0ea0ee065441be78b7, entries=150, sequenceid=575, filesize=11.9 K 2023-04-19 07:56:08,680 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 54144 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:56:08,680 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/C is initiating minor compaction (all files) 2023-04-19 07:56:08,680 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/C in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:08,680 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/9e7f4ac311544aaa9998c7f96a86e9b5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a1f6d12bd18348bb8f5bdfd045062702, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/f6ba06c380a4426fa5733906aeaa746d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/6ead41732a8249ef9767e47d083026b4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/5b202ca9b6d14e0a81b29f007e7db001] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=52.9 K 2023-04-19 07:56:08,681 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 9e7f4ac311544aaa9998c7f96a86e9b5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681890966049 2023-04-19 07:56:08,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/46ff68515b7f4c0d8075db6600e61e5b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/46ff68515b7f4c0d8075db6600e61e5b 2023-04-19 07:56:08,681 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting a1f6d12bd18348bb8f5bdfd045062702, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1681890966353 2023-04-19 07:56:08,681 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting f6ba06c380a4426fa5733906aeaa746d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681890967304 2023-04-19 07:56:08,682 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 6ead41732a8249ef9767e47d083026b4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1681890967936 2023-04-19 07:56:08,682 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 5b202ca9b6d14e0a81b29f007e7db001, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1681890968563 2023-04-19 07:56:08,685 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/46ff68515b7f4c0d8075db6600e61e5b, entries=150, sequenceid=575, filesize=11.9 K 2023-04-19 07:56:08,686 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/d4f911fb532b4fe19213914f1bc5ea38 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/d4f911fb532b4fe19213914f1bc5ea38 2023-04-19 07:56:08,690 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/d4f911fb532b4fe19213914f1bc5ea38, entries=150, sequenceid=575, filesize=11.9 K 2023-04-19 07:56:08,690 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~101.34 KB/103776, currentSize=53.67 KB/54960 for 199c6b7f1f3a9353227e2a51c7965b15 in 80ms, sequenceid=575, compaction requested=false 2023-04-19 07:56:08,690 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:08,690 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:08,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:08,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:08,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:08,691 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=60.38 KB heapSize=165.84 KB 2023-04-19 07:56:08,691 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:08,691 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:08,691 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:08,691 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:08,691 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:08,691 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:08,691 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,697 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#C#compaction#452 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:08,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:08,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:08,702 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,702 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,703 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,703 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:08,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6edf5273[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:08,705 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:08,705 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,706 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,706 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:08,706 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:08,706 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:08,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:08,707 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,707 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,707 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:08,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:08,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:08,711 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:08,711 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:08,711 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:08,711 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:08,711 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:08,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,714 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,715 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:08,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:08,718 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:08,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:08,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:08,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:08,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,719 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,719 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,719 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:08,719 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:08,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:08,722 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/478136419d87497f84c67f3cfcdf70ac as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/478136419d87497f84c67f3cfcdf70ac 2023-04-19 07:56:08,727 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/C of 199c6b7f1f3a9353227e2a51c7965b15 into 478136419d87497f84c67f3cfcdf70ac(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:08,727 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:08,727 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/C, priority=11, startTime=1681890968610; duration=0sec 2023-04-19 07:56:08,727 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:08,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891028733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890968721, max timestamp=1681890968736Num uniques 859; 2023-04-19 07:56:08,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:56:08,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890968721, max timestamp=1681890968736Num uniques 859; 2023-04-19 07:56:08,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681890968721, max timestamp=1681890968736Num uniques 859; 2023-04-19 07:56:08,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:56:08,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:56:08,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891028781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891028781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891028782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,819 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:08,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:08,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:08,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:08,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:08,822 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:08,822 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:08,822 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:08,823 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:08,823 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,823 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,824 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:08,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:08,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:08,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,825 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:08,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:08,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:08,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:08,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:08,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:08,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:08,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:08,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:08,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:08,826 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:08,826 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:08,826 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:08,826 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:08,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:08,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:08,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:08,827 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:08,828 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,828 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,830 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,830 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:08,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:08,830 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:08,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f4ffb15[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:08,830 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,831 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,831 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,831 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:08,831 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:08,831 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:08,831 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:08,834 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:08,834 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:08,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,834 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:08,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:08,836 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:08,837 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:08,837 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:08,837 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:08,837 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:08,838 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,838 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:08,838 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,839 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,840 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,842 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,843 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,843 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,843 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,843 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,843 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,843 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,844 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:08,844 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:08,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:08,844 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:08,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:08,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:08,844 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:08,844 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:08,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:08,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891028838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891028888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891028890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:08,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891028891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,944 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:08,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:08,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:08,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:08,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:08,948 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:08,948 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:08,948 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:08,948 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:08,948 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,948 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,949 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:08,949 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:08,950 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,950 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:08,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:08,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:08,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:08,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:08,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:08,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:08,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:08,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:08,951 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:08,951 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:08,952 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:08,952 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:08,952 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:08,952 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:08,952 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,952 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:08,952 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:08,954 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,954 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,954 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,955 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:08,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:08,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2964f6d1[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:08,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,958 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,958 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:08,958 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:08,959 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:08,959 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,959 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:08,960 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,960 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:08,960 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,960 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:08,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:08,962 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:08,962 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:08,962 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:08,963 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:08,963 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:08,964 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,964 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:08,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:08,964 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:08,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,965 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,966 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:08,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:08,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,971 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:08,971 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:08,972 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,972 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:08,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:08,972 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:08,972 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:08,972 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,972 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:08,972 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:08,972 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:08,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:08,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:08,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:08,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:08,973 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:08,973 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:08,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:09,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:09,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891029047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,073 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:09,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:09,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:09,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:09,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:09,076 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:09,076 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:09,076 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:09,076 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:09,077 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,077 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,078 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:09,078 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:09,078 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,078 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:09,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:09,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:09,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:09,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:09,079 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:09,079 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:09,079 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:09,080 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:09,080 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:09,080 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:09,080 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:09,080 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:09,080 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:09,080 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:09,080 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:09,080 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,080 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:09,080 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:09,084 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,084 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:09,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:09,084 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,084 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:09,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:09,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:09,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:09,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:09,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47ee9a0c[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:09,086 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:09,086 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,088 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,088 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:09,088 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:09,088 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,088 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:09,088 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:09,089 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,089 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:09,089 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:09,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:09,090 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:09,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:09,092 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:09,092 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:09,092 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:09,092 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:09,092 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:09,094 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,094 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,094 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:09,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:09,094 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:09,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,096 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:09,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:09,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891029095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:09,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891029095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:09,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891029095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,102 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:09,102 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,102 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:09,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:09,102 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:09,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:09,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:09,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:09,102 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,102 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,102 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,102 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,102 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,102 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:09,103 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:09,102 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:09,126 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/bace4c8adecc4ab6a20be6d096e20192 2023-04-19 07:56:09,203 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:09,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:09,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:09,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:09,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:09,206 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:09,206 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:09,206 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:09,206 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:09,207 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,207 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,209 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:09,209 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:09,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:09,210 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,210 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:09,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:09,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:09,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:09,211 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:09,211 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:09,211 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:09,211 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:09,211 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:09,211 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:09,211 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:09,212 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:09,212 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:09,212 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:09,212 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:09,212 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,212 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:09,212 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:09,213 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,214 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,214 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:09,214 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:09,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:09,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53b60448[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:09,216 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:09,216 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,217 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,217 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:09,217 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:09,218 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:09,218 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,218 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:09,219 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:09,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:09,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:09,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:09,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:09,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:09,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:09,221 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:09,221 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:09,221 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:09,221 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:09,222 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:09,225 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,225 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:09,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,225 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:09,226 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:09,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:09,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,227 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:09,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,230 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,230 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,230 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,230 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,230 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,230 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,230 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,230 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:09,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:09,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:09,230 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:09,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:09,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:09,230 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:09,230 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:09,230 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:09,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:09,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:09,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:09,330 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:09,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:09,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:09,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:09,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:09,334 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:09,334 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:09,334 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:09,334 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:09,335 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,335 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,336 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:09,336 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:09,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:09,336 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,336 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:09,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:09,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:09,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:09,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:09,338 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:09,338 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:09,338 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:09,338 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:09,338 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:09,338 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:09,338 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:09,338 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:09,338 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:09,338 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:09,338 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:09,338 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,338 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:09,338 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:09,340 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:09,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,340 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:09,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:09,342 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:09,342 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@371da78[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:09,344 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,344 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:09,344 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:09,344 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:09,344 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,344 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:09,345 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,345 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:09,346 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:09,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:09,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:09,346 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:09,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:09,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:09,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:09,348 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:09,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:09,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:09,348 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:09,348 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:09,350 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,350 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,350 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:09,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:09,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:09,350 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:09,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:09,351 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,354 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,354 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,354 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,354 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,354 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:09,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:09,354 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:09,354 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:09,355 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:09,355 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:09,354 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:09,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:09,354 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,354 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:09,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:09,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:09,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:09,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891029351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:09,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891029401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:09,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891029401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:09,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891029402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,455 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:09,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:09,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:09,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:09,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:09,458 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:09,458 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:09,458 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:09,458 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:09,459 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,459 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,461 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,461 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:09,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:09,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:09,461 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,462 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:09,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:09,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:09,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:09,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:09,463 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:09,463 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:09,463 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:09,463 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:09,465 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,465 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:09,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:09,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,467 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:09,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:09,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:09,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:09,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:09,468 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:09,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dad66b9[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:09,468 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,469 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:09,470 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:09,470 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:09,470 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,470 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:09,474 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:09,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:09,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:09,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:09,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:09,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:09,477 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:09,477 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:09,477 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:09,477 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:09,477 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:09,478 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,478 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:09,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,479 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:09,479 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:09,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,480 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:09,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,484 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:09,484 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:09,484 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:09,484 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:09,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:09,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:09,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:09,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:09,484 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,485 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,485 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,485 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,485 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:09,485 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:09,485 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:09,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:09,485 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,535 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/7d9cc1641ef24763b5401fe3007ab9c4 2023-04-19 07:56:09,545 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/a0d623c5c960455aa16fc5d64753eb0b 2023-04-19 07:56:09,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/bace4c8adecc4ab6a20be6d096e20192 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/bace4c8adecc4ab6a20be6d096e20192 2023-04-19 07:56:09,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/bace4c8adecc4ab6a20be6d096e20192, entries=250, sequenceid=603, filesize=16.7 K 2023-04-19 07:56:09,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/7d9cc1641ef24763b5401fe3007ab9c4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/7d9cc1641ef24763b5401fe3007ab9c4 2023-04-19 07:56:09,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/7d9cc1641ef24763b5401fe3007ab9c4, entries=150, sequenceid=603, filesize=11.9 K 2023-04-19 07:56:09,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/a0d623c5c960455aa16fc5d64753eb0b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a0d623c5c960455aa16fc5d64753eb0b 2023-04-19 07:56:09,562 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a0d623c5c960455aa16fc5d64753eb0b, entries=150, sequenceid=603, filesize=11.9 K 2023-04-19 07:56:09,562 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~165.80 KB/169776, currentSize=127.47 KB/130530 for 199c6b7f1f3a9353227e2a51c7965b15 in 871ms, sequenceid=603, compaction requested=true 2023-04-19 07:56:09,562 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:09,563 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:09,563 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:09,563 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:09,563 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:09,563 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:09,564 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42154 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:09,564 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:09,564 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/A is initiating minor compaction (all files) 2023-04-19 07:56:09,564 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/A in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:09,564 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/B is initiating minor compaction (all files) 2023-04-19 07:56:09,564 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/B in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:09,564 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2d3b1696ef6a48fb98076c6c0210a27d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/46ff68515b7f4c0d8075db6600e61e5b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/7d9cc1641ef24763b5401fe3007ab9c4] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=36.4 K 2023-04-19 07:56:09,565 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 2d3b1696ef6a48fb98076c6c0210a27d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1681890967936 2023-04-19 07:56:09,565 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 46ff68515b7f4c0d8075db6600e61e5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681890968602 2023-04-19 07:56:09,565 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7d9cc1641ef24763b5401fe3007ab9c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681890968652 2023-04-19 07:56:09,564 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/71ef49dea68448e2b7301533e5fd3079, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/b535673690244e0ea0ee065441be78b7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/bace4c8adecc4ab6a20be6d096e20192] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=41.2 K 2023-04-19 07:56:09,566 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 71ef49dea68448e2b7301533e5fd3079, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1681890967936 2023-04-19 07:56:09,566 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting b535673690244e0ea0ee065441be78b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681890968602 2023-04-19 07:56:09,567 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting bace4c8adecc4ab6a20be6d096e20192, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681890968652 2023-04-19 07:56:09,571 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#B#compaction#456 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:09,575 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#A#compaction#457 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:09,585 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:09,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:09,586 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/e40f0880d1634c91bcbcded9c849e213 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/e40f0880d1634c91bcbcded9c849e213 2023-04-19 07:56:09,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:09,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:09,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:09,589 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:09,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:09,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:09,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:09,590 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,590 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,591 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:09,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:09,591 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,591 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/A of 199c6b7f1f3a9353227e2a51c7965b15 into e40f0880d1634c91bcbcded9c849e213(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:09,591 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:09,591 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/A, priority=13, startTime=1681890969563; duration=0sec 2023-04-19 07:56:09,591 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:09,592 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,592 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,592 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:09,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:09,593 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:09,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:09,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:09,593 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/C is initiating minor compaction (all files) 2023-04-19 07:56:09,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:09,593 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/C in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:09,593 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/478136419d87497f84c67f3cfcdf70ac, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/d4f911fb532b4fe19213914f1bc5ea38, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a0d623c5c960455aa16fc5d64753eb0b] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=36.4 K 2023-04-19 07:56:09,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:09,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:09,593 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:09,593 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:09,593 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 478136419d87497f84c67f3cfcdf70ac, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1681890967936 2023-04-19 07:56:09,593 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:09,594 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:09,594 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:09,594 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:09,594 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting d4f911fb532b4fe19213914f1bc5ea38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681890968602 2023-04-19 07:56:09,594 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:09,594 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-19 07:56:09,594 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting a0d623c5c960455aa16fc5d64753eb0b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681890968652 2023-04-19 07:56:09,594 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:09,594 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:09,594 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:09,594 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:09,595 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:09,595 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:09,605 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#C#compaction#459 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:09,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:09,867 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:09,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:09,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891029909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:09,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891029909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:09,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891029909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:09,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891029913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:09,981 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/2516cd48867344c385b9b5d56e160899 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2516cd48867344c385b9b5d56e160899 2023-04-19 07:56:09,985 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/B of 199c6b7f1f3a9353227e2a51c7965b15 into 2516cd48867344c385b9b5d56e160899(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:09,985 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:09,985 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/B, priority=13, startTime=1681890969563; duration=0sec 2023-04-19 07:56:09,985 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:10,014 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/fd0297c71067406ab4ed7d74ff59f8e8 2023-04-19 07:56:10,020 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/7a114ccddc474f37b3768c6916ecaa05 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/7a114ccddc474f37b3768c6916ecaa05 2023-04-19 07:56:10,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:10,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891030015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,026 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/C of 199c6b7f1f3a9353227e2a51c7965b15 into 7a114ccddc474f37b3768c6916ecaa05(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:10,027 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:10,027 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/C, priority=13, startTime=1681890969563; duration=0sec 2023-04-19 07:56:10,027 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:10,027 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/bf676ce9b6c54048848d8fe08e977606 2023-04-19 07:56:10,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:10,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891030023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:10,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891030225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:10,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891030232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,437 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=639 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/be10b73006914c5297b62f691e699a83 2023-04-19 07:56:10,441 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/fd0297c71067406ab4ed7d74ff59f8e8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/fd0297c71067406ab4ed7d74ff59f8e8 2023-04-19 07:56:10,444 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/fd0297c71067406ab4ed7d74ff59f8e8, entries=150, sequenceid=639, filesize=11.9 K 2023-04-19 07:56:10,445 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/bf676ce9b6c54048848d8fe08e977606 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/bf676ce9b6c54048848d8fe08e977606 2023-04-19 07:56:10,448 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/bf676ce9b6c54048848d8fe08e977606, entries=150, sequenceid=639, filesize=11.9 K 2023-04-19 07:56:10,449 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/be10b73006914c5297b62f691e699a83 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/be10b73006914c5297b62f691e699a83 2023-04-19 07:56:10,453 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/be10b73006914c5297b62f691e699a83, entries=150, sequenceid=639, filesize=11.9 K 2023-04-19 07:56:10,454 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 199c6b7f1f3a9353227e2a51c7965b15 in 860ms, sequenceid=639, compaction requested=false 2023-04-19 07:56:10,454 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:10,454 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:10,454 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:10,454 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:10,454 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,454 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:10,454 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:10,456 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e20b3fa[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:10,457 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:10,458 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,459 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,459 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:10,459 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:10,459 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:10,459 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,459 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:10,460 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:10,460 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:10,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:10,462 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:10,462 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:10,462 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:10,462 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:10,462 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:10,465 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,465 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,465 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:10,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,465 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:10,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,466 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:10,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:10,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:10,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:10,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:10,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,469 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:10,469 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:10,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:10,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:10,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:10,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:10,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:10,534 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 07:56:10,534 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:10,534 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:10,534 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:10,534 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:10,534 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:10,534 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:10,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/6692ec19816b41b88638ae866c21d686 2023-04-19 07:56:10,569 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:10,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:10,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:10,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:10,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:10,573 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:10,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:10,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:10,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:10,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,573 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,574 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,574 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:10,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:10,575 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,575 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:10,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:10,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:10,576 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:10,576 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:10,576 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:10,576 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:10,577 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:10,577 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:10,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,578 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,579 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:10,580 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:10,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dd76432[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:10,580 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,582 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:10,582 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:10,582 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:10,582 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:10,583 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,584 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:10,584 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,584 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:10,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:10,586 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:10,586 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:10,586 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:10,586 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:10,586 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:10,588 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,588 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,588 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:10,588 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,590 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:10,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:10,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:10,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:10,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:10,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:10,592 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:10,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:10,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:10,592 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:10,592 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:10,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:10,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:10,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891030585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:10,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891030587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:10,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37924 deadline: 1681891030652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,660 DEBUG [Thread-2959] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8185 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:56:10,693 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:10,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:10,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:10,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:10,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:10,696 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:10,696 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:10,696 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:10,696 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:10,696 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,696 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,698 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,698 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:10,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:10,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:10,699 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,699 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:10,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:10,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:10,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891030694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:10,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:10,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:10,700 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:10,700 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:10,700 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:10,700 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:10,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:10,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891030696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,702 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,702 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,702 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,702 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:10,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5078d7ea[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:10,704 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:10,704 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,705 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,705 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:10,705 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:10,705 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:10,705 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,705 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:10,706 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,706 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:10,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,706 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:10,706 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:10,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:10,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:10,709 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:10,709 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:10,709 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:10,709 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:10,709 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:10,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,719 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:10,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,720 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,722 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,723 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:10,723 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:10,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:10,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:10,723 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:10,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:10,724 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,724 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:10,724 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:10,724 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:10,724 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,724 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,724 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,724 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,724 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:10,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:10,824 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:10,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:10,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:10,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:10,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:10,828 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:10,828 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:10,828 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:10,828 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:10,828 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,828 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,829 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:10,829 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:10,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:10,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,831 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,831 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:10,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:10,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:10,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:10,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:10,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:10,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:10,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:10,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:10,832 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:10,832 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:10,832 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:10,832 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:10,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:10,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:10,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:10,832 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:10,834 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,834 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,834 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,834 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:10,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:10,835 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:10,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3606ebf2[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:10,836 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,837 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:10,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:10,837 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:10,838 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,838 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:10,839 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:10,839 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,839 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,839 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:10,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:10,841 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:10,841 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:10,841 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:10,841 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:10,841 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:10,844 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,844 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,844 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:10,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:10,845 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,846 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,848 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,848 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,848 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,848 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:10,848 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:10,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:10,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:10,848 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:10,848 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:10,848 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:10,849 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:10,848 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,849 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,849 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:10,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:10,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891030901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:10,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891030903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:10,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891030918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:10,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891030919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,949 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:10,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:10,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:10,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:10,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:10,952 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:10,952 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:10,953 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:10,953 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:10,953 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,953 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,955 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:10,955 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:10,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:10,955 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/7a22bd66598f437b96599a6db54ed4d8 2023-04-19 07:56:10,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,956 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,956 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:10,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:10,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:10,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:10,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:10,957 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:10,957 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:10,957 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:10,957 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:10,958 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:10,958 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:10,958 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:10,958 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:10,958 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:10,958 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:10,958 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:10,958 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,958 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:10,958 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:10,959 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,959 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,960 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,960 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:10,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d8221ab[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:10,962 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:10,962 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,964 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,965 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:10,965 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:10,965 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:10,965 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,965 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:10,966 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:10,966 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,966 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,966 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/1f29e0f26c484f5ab3f8b4e5d8022735 2023-04-19 07:56:10,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:10,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:10,970 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:10,970 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:10,970 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:10,970 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:10,970 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:10,973 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,973 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:10,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,974 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,974 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/6692ec19816b41b88638ae866c21d686 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/6692ec19816b41b88638ae866c21d686 2023-04-19 07:56:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:10,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:10,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:10,974 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:10,976 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:10,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:10,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:10,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:10,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:10,979 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/6692ec19816b41b88638ae866c21d686, entries=200, sequenceid=653, filesize=14.3 K 2023-04-19 07:56:10,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:10,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:10,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:10,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:10,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:10,979 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:10,980 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:10,980 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:10,979 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:10,980 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/7a22bd66598f437b96599a6db54ed4d8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/7a22bd66598f437b96599a6db54ed4d8 2023-04-19 07:56:10,980 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:10,980 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,980 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:10,980 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:10,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/7a22bd66598f437b96599a6db54ed4d8, entries=150, sequenceid=653, filesize=11.9 K 2023-04-19 07:56:10,985 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/1f29e0f26c484f5ab3f8b4e5d8022735 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/1f29e0f26c484f5ab3f8b4e5d8022735 2023-04-19 07:56:10,989 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/1f29e0f26c484f5ab3f8b4e5d8022735, entries=150, sequenceid=653, filesize=11.9 K 2023-04-19 07:56:10,990 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 199c6b7f1f3a9353227e2a51c7965b15 in 456ms, sequenceid=653, compaction requested=true 2023-04-19 07:56:10,990 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:10,990 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:10,991 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:10,991 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:10,991 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:10,991 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:10,992 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39816 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:10,992 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/A is initiating minor compaction (all files) 2023-04-19 07:56:10,992 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/A in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:10,992 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:10,992 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/e40f0880d1634c91bcbcded9c849e213, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/fd0297c71067406ab4ed7d74ff59f8e8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/6692ec19816b41b88638ae866c21d686] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=38.9 K 2023-04-19 07:56:10,992 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/B is initiating minor compaction (all files) 2023-04-19 07:56:10,992 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/B in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:10,992 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2516cd48867344c385b9b5d56e160899, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/bf676ce9b6c54048848d8fe08e977606, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/7a22bd66598f437b96599a6db54ed4d8] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=36.5 K 2023-04-19 07:56:10,992 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting e40f0880d1634c91bcbcded9c849e213, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681890968652 2023-04-19 07:56:10,993 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 2516cd48867344c385b9b5d56e160899, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681890968652 2023-04-19 07:56:10,993 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting fd0297c71067406ab4ed7d74ff59f8e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1681890968755 2023-04-19 07:56:10,993 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting bf676ce9b6c54048848d8fe08e977606, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1681890968755 2023-04-19 07:56:10,993 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 6692ec19816b41b88638ae866c21d686, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1681890969884 2023-04-19 07:56:10,994 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7a22bd66598f437b96599a6db54ed4d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1681890969884 2023-04-19 07:56:11,003 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#A#compaction#465 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:11,007 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#B#compaction#466 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:11,080 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:11,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:11,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:11,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:11,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:11,083 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:11,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:11,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:11,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:11,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:11,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:11,085 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:11,085 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:11,085 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,085 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:11,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:11,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:11,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:11,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:11,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:11,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:11,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:11,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:11,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:11,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:11,086 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:11,086 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:11,086 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:11,086 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-19 07:56:11,087 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:11,087 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:11,087 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:11,087 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:11,087 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:11,087 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:11,094 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/d8900e45edb24209a98b29eb51bc94af 2023-04-19 07:56:11,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:11,209 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:11,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:11,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891031240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:11,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891031243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:11,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891031347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:11,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891031349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,419 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/f0ad67f4654c41318a8ec656228de284 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/f0ad67f4654c41318a8ec656228de284 2023-04-19 07:56:11,423 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/A of 199c6b7f1f3a9353227e2a51c7965b15 into f0ad67f4654c41318a8ec656228de284(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:11,423 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:11,423 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/A, priority=13, startTime=1681890970990; duration=0sec 2023-04-19 07:56:11,423 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:11,423 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:11,424 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:11,424 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/C is initiating minor compaction (all files) 2023-04-19 07:56:11,424 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/C in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:11,424 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/7a114ccddc474f37b3768c6916ecaa05, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/be10b73006914c5297b62f691e699a83, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/1f29e0f26c484f5ab3f8b4e5d8022735] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=36.5 K 2023-04-19 07:56:11,425 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 7a114ccddc474f37b3768c6916ecaa05, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681890968652 2023-04-19 07:56:11,425 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting be10b73006914c5297b62f691e699a83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=639, earliestPutTs=1681890968755 2023-04-19 07:56:11,426 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 1f29e0f26c484f5ab3f8b4e5d8022735, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1681890969884 2023-04-19 07:56:11,427 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/0ab8713a06c047f49835d5a41a211b8a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/0ab8713a06c047f49835d5a41a211b8a 2023-04-19 07:56:11,431 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/B of 199c6b7f1f3a9353227e2a51c7965b15 into 0ab8713a06c047f49835d5a41a211b8a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:11,431 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:11,431 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/B, priority=13, startTime=1681890970991; duration=0sec 2023-04-19 07:56:11,431 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:11,432 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#C#compaction#469 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:11,506 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/fb4f55c9fea14674b5a3e5eb2e9753a1 2023-04-19 07:56:11,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:11,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891031550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:11,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891031555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,840 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/0e9ab4bca2fd46b9a2b83eea00369b52 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0e9ab4bca2fd46b9a2b83eea00369b52 2023-04-19 07:56:11,844 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/C of 199c6b7f1f3a9353227e2a51c7965b15 into 0e9ab4bca2fd46b9a2b83eea00369b52(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:11,844 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:11,844 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/C, priority=13, startTime=1681890970991; duration=0sec 2023-04-19 07:56:11,845 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:11,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:11,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891031856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:11,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891031863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,916 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/5c97266aefb84ea8bd5263fcf5013fa0 2023-04-19 07:56:11,920 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/d8900e45edb24209a98b29eb51bc94af as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d8900e45edb24209a98b29eb51bc94af 2023-04-19 07:56:11,923 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d8900e45edb24209a98b29eb51bc94af, entries=150, sequenceid=673, filesize=11.9 K 2023-04-19 07:56:11,924 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/fb4f55c9fea14674b5a3e5eb2e9753a1 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/fb4f55c9fea14674b5a3e5eb2e9753a1 2023-04-19 07:56:11,928 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/fb4f55c9fea14674b5a3e5eb2e9753a1, entries=150, sequenceid=673, filesize=11.9 K 2023-04-19 07:56:11,929 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/5c97266aefb84ea8bd5263fcf5013fa0 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/5c97266aefb84ea8bd5263fcf5013fa0 2023-04-19 07:56:11,932 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/5c97266aefb84ea8bd5263fcf5013fa0, entries=150, sequenceid=673, filesize=11.9 K 2023-04-19 07:56:11,933 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 199c6b7f1f3a9353227e2a51c7965b15 in 846ms, sequenceid=673, compaction requested=false 2023-04-19 07:56:11,933 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:11,933 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:11,933 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:11,933 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:11,933 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,933 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:11,933 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:11,934 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:11,935 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:11,935 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:11,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:11,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b9232dd[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:11,936 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:11,936 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:11,938 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:11,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:11,938 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:11,938 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:11,938 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:11,938 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,938 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:11,939 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:11,939 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,939 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:11,939 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:11,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:11,942 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:11,942 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:11,942 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:11,942 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:11,942 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:11,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:11,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:11,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:11,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:11,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:11,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:11,944 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:11,945 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:11,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,952 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:11,952 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:11,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:11,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:11,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:11,952 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:11,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:11,953 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:11,953 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:11,952 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:11,952 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:11,953 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:11,953 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:11,953 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:11,953 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:11,953 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:11,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:12,053 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:12,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:12,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:12,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:12,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:12,056 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:12,056 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:12,056 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:12,056 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:12,057 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:12,057 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:12,058 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:12,058 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:12,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:12,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:12,059 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,059 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:12,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:12,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:12,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:12,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:12,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:12,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:12,060 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:12,060 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:12,060 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:12,060 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:12,060 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:12,060 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:12,060 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:12,060 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 07:56:12,060 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:12,061 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:12,061 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:12,061 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:12,061 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:12,061 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:12,079 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/fc1a0eded4114d1a9427f4a2205c30ff 2023-04-19 07:56:12,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:12,366 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:12,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:12,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891032418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:12,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891032422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,492 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/502fb839661e4bb8b35739211a7f1045 2023-04-19 07:56:12,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:12,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891032524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:12,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891032528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:12,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891032726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:12,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891032732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,903 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/6f985faca6fa4943b77f0ddc0a0e9c34 2023-04-19 07:56:12,906 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/fc1a0eded4114d1a9427f4a2205c30ff as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/fc1a0eded4114d1a9427f4a2205c30ff 2023-04-19 07:56:12,910 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/fc1a0eded4114d1a9427f4a2205c30ff, entries=150, sequenceid=688, filesize=11.9 K 2023-04-19 07:56:12,910 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/502fb839661e4bb8b35739211a7f1045 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/502fb839661e4bb8b35739211a7f1045 2023-04-19 07:56:12,914 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/502fb839661e4bb8b35739211a7f1045, entries=150, sequenceid=688, filesize=11.9 K 2023-04-19 07:56:12,915 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/6f985faca6fa4943b77f0ddc0a0e9c34 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/6f985faca6fa4943b77f0ddc0a0e9c34 2023-04-19 07:56:12,919 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/6f985faca6fa4943b77f0ddc0a0e9c34, entries=150, sequenceid=688, filesize=11.9 K 2023-04-19 07:56:12,919 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 199c6b7f1f3a9353227e2a51c7965b15 in 859ms, sequenceid=688, compaction requested=true 2023-04-19 07:56:12,919 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:12,919 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:12,919 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:12,919 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:12,919 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,920 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:12,920 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:12,921 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,921 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:12,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:12,921 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:12,921 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:12,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:12,923 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:12,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc37fc2[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:12,923 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:12,924 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:12,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:12,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:12,924 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:12,924 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,924 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:12,929 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,929 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:12,929 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:12,929 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:12,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:12,932 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:12,932 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:12,932 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:12,932 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:12,932 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:12,933 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:12,933 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:12,933 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:12,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:12,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890972416, max timestamp=1681890972933Num uniques 859; 2023-04-19 07:56:12,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:12,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:12,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890972416, max timestamp=1681890972933Num uniques 859; 2023-04-19 07:56:12,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:12,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890972416, max timestamp=1681890972933Num uniques 859; 2023-04-19 07:56:12,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:12,936 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:12,937 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=26.84 KB heapSize=66.23 KB 2023-04-19 07:56:12,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:12,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:12,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:12,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:12,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:12,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:12,938 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:12,939 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:12,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:12,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:12,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:12,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:12,943 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:12,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:12,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:12,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:12,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:12,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:12,943 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:12,943 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:12,943 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:12,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:12,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:12,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:13,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890972970, max timestamp=1681890973013Num uniques 859; 2023-04-19 07:56:13,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890972970, max timestamp=1681890973013Num uniques 859; 2023-04-19 07:56:13,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681890972970, max timestamp=1681890973013Num uniques 859; 2023-04-19 07:56:13,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:56:13,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:56:13,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-19 07:56:13,043 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:13,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:13,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:13,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:13,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:13,047 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:13,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:13,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:13,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:13,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,048 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,049 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:13,049 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:13,049 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,049 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:13,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:13,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:13,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:13,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:13,050 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:13,051 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:13,051 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:13,051 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:13,051 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:13,051 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:13,051 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:13,051 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:13,051 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:13,051 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,051 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:13,051 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:13,053 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,053 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:13,054 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,054 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:13,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c1b42f3[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:13,055 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:13,055 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,056 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,057 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,057 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:13,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:13,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:13,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:13,058 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:13,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:13,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:13,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:13,061 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:13,061 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:13,061 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:13,061 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:13,061 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:13,062 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,062 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,062 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:13,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:13,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:13,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,063 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,065 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:13,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890973049, max timestamp=1681890973064Num uniques 1068; 2023-04-19 07:56:13,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-19 07:56:13,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890973049, max timestamp=1681890973064Num uniques 1068; 2023-04-19 07:56:13,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-19 07:56:13,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681890973049, max timestamp=1681890973064Num uniques 1068; 2023-04-19 07:56:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-19 07:56:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,070 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:13,070 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:13,070 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:13,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:13,070 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:13,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:13,070 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:13,070 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:13,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:13,070 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:13,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:13,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681890973078, max timestamp=1681890973110Num uniques 1127; 2023-04-19 07:56:13,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-19 07:56:13,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681890973078, max timestamp=1681890973110Num uniques 1127; 2023-04-19 07:56:13,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1681890973078, max timestamp=1681890973110Num uniques 1127; 2023-04-19 07:56:13,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-19 07:56:13,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-19 07:56:13,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891033146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891033147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891033147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891033156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,171 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:13,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:13,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:13,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:13,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:13,173 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:13,173 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:13,174 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:13,174 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:13,174 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,174 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,175 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:13,175 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:13,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:13,175 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,175 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:13,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:13,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:13,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:13,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:13,176 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:13,176 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:13,177 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:13,177 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:13,177 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:13,177 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:13,178 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,178 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:13,179 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:13,179 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:13,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:13,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6008bba2[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:13,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:13,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,181 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:13,181 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,181 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:13,181 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:13,184 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:13,184 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:13,184 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,184 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:13,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:13,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:13,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:13,187 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:13,187 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:13,187 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:13,187 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:13,187 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:13,188 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,189 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,189 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:13,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:13,189 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:13,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,190 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:13,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:13,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:13,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:13,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:13,195 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,196 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,196 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:13,196 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:13,196 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:13,196 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:13,196 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,196 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,196 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,196 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891033257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891033257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891033257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891033265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,296 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:13,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:13,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:13,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:13,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:13,300 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:13,300 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:13,300 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:13,300 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:13,300 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,300 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,302 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:13,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:13,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:13,303 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,303 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:13,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:13,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:13,305 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:13,306 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:13,306 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:13,306 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:13,306 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:13,306 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:13,306 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:13,306 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:13,306 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,306 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:13,306 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:13,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:13,307 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,308 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,308 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:13,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:13,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:13,310 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:13,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1af679a4[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:13,310 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,311 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:13,311 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,311 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:13,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:13,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:13,313 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,313 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:13,313 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:13,313 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:13,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:13,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:13,316 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:13,316 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:13,316 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:13,316 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:13,316 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:13,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:13,321 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:13,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:13,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:13,322 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,324 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,327 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,327 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:13,327 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:13,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:13,327 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:13,327 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,327 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,327 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,327 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,327 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:13,327 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:13,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:13,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:13,328 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:13,328 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:13,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=710 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/c1a9ff30ece24e02a3624496c004d96c 2023-04-19 07:56:13,356 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=710 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/53f49b4bdbaa44e387d9e08ef27137ca 2023-04-19 07:56:13,366 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=710 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/77347b6a4f144e6f8401a03da8847246 2023-04-19 07:56:13,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/c1a9ff30ece24e02a3624496c004d96c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/c1a9ff30ece24e02a3624496c004d96c 2023-04-19 07:56:13,375 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/c1a9ff30ece24e02a3624496c004d96c, entries=250, sequenceid=710, filesize=16.7 K 2023-04-19 07:56:13,375 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/53f49b4bdbaa44e387d9e08ef27137ca as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/53f49b4bdbaa44e387d9e08ef27137ca 2023-04-19 07:56:13,379 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/53f49b4bdbaa44e387d9e08ef27137ca, entries=150, sequenceid=710, filesize=11.9 K 2023-04-19 07:56:13,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/77347b6a4f144e6f8401a03da8847246 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/77347b6a4f144e6f8401a03da8847246 2023-04-19 07:56:13,382 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/77347b6a4f144e6f8401a03da8847246, entries=150, sequenceid=710, filesize=11.9 K 2023-04-19 07:56:13,383 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=154.31 KB/158010 for 199c6b7f1f3a9353227e2a51c7965b15 in 446ms, sequenceid=710, compaction requested=true 2023-04-19 07:56:13,383 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:13,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:13,383 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:56:13,383 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:56:13,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:13,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:13,386 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:56:13,386 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:56:13,386 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/A is initiating minor compaction (all files) 2023-04-19 07:56:13,386 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/B is initiating minor compaction (all files) 2023-04-19 07:56:13,386 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/A in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:13,386 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/B in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:13,386 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/0ab8713a06c047f49835d5a41a211b8a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/fb4f55c9fea14674b5a3e5eb2e9753a1, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/502fb839661e4bb8b35739211a7f1045, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/53f49b4bdbaa44e387d9e08ef27137ca] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=48.5 K 2023-04-19 07:56:13,386 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/f0ad67f4654c41318a8ec656228de284, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d8900e45edb24209a98b29eb51bc94af, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/fc1a0eded4114d1a9427f4a2205c30ff, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/c1a9ff30ece24e02a3624496c004d96c] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=53.3 K 2023-04-19 07:56:13,386 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 0ab8713a06c047f49835d5a41a211b8a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1681890969884 2023-04-19 07:56:13,386 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting f0ad67f4654c41318a8ec656228de284, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1681890969884 2023-04-19 07:56:13,387 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting d8900e45edb24209a98b29eb51bc94af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1681890970582 2023-04-19 07:56:13,387 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting fb4f55c9fea14674b5a3e5eb2e9753a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1681890970582 2023-04-19 07:56:13,387 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting fc1a0eded4114d1a9427f4a2205c30ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1681890971222 2023-04-19 07:56:13,387 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 502fb839661e4bb8b35739211a7f1045, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1681890971222 2023-04-19 07:56:13,387 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting c1a9ff30ece24e02a3624496c004d96c, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1681890972416 2023-04-19 07:56:13,387 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 53f49b4bdbaa44e387d9e08ef27137ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1681890972421 2023-04-19 07:56:13,393 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#A#compaction#477 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:13,395 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#B#compaction#478 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:13,400 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/ce40183fd04e45cdb82e2e129d34d6a7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/ce40183fd04e45cdb82e2e129d34d6a7 2023-04-19 07:56:13,404 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/A of 199c6b7f1f3a9353227e2a51c7965b15 into ce40183fd04e45cdb82e2e129d34d6a7(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:13,404 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:13,404 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/A, priority=12, startTime=1681890973383; duration=0sec 2023-04-19 07:56:13,404 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:13,404 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:56:13,406 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:56:13,406 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/C is initiating minor compaction (all files) 2023-04-19 07:56:13,406 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/C in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:13,406 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0e9ab4bca2fd46b9a2b83eea00369b52, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/5c97266aefb84ea8bd5263fcf5013fa0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/6f985faca6fa4943b77f0ddc0a0e9c34, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/77347b6a4f144e6f8401a03da8847246] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=48.5 K 2023-04-19 07:56:13,406 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 0e9ab4bca2fd46b9a2b83eea00369b52, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1681890969884 2023-04-19 07:56:13,407 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 5c97266aefb84ea8bd5263fcf5013fa0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1681890970582 2023-04-19 07:56:13,407 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 6f985faca6fa4943b77f0ddc0a0e9c34, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1681890971222 2023-04-19 07:56:13,407 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 77347b6a4f144e6f8401a03da8847246, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1681890972421 2023-04-19 07:56:13,412 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#C#compaction#479 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:13,421 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/4590f1e1ffa443deaa3a589ca92dccf0 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/4590f1e1ffa443deaa3a589ca92dccf0 2023-04-19 07:56:13,425 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/C of 199c6b7f1f3a9353227e2a51c7965b15 into 4590f1e1ffa443deaa3a589ca92dccf0(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:13,425 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:13,425 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/C, priority=12, startTime=1681890973384; duration=0sec 2023-04-19 07:56:13,425 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:13,428 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:13,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:13,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:13,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:13,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:13,431 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:13,431 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:13,431 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:13,431 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:13,431 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,431 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,432 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,432 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:13,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:13,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:13,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:13,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:13,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:13,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:13,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:13,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:13,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:13,438 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:13,438 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:13,438 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:13,440 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:13,440 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:13,440 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:13,440 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:13,440 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=154.31 KB heapSize=429 KB 2023-04-19 07:56:13,440 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:13,440 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:13,440 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:13,440 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:13,440 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:13,440 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:13,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:13,473 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:13,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891033475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891033476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891033482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891033483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891033584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891033584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891033590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891033590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891033789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891033790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891033796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:13,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891033797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:13,821 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/f690ccbb7cb34ecf8345fa7b7f62e645 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/f690ccbb7cb34ecf8345fa7b7f62e645 2023-04-19 07:56:13,825 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/B of 199c6b7f1f3a9353227e2a51c7965b15 into f690ccbb7cb34ecf8345fa7b7f62e645(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:13,825 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:13,825 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/B, priority=12, startTime=1681890973383; duration=0sec 2023-04-19 07:56:13,825 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:13,859 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/beb44b2152cf40579c71875b282b5750 2023-04-19 07:56:14,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:14,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891034095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:14,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891034096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:14,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891034102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:14,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891034103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,269 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/627039e1991f45038d28096a018bfabf 2023-04-19 07:56:14,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:14,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891034606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:14,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891034607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:14,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891034611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:14,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891034614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,681 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=783 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/0b9152b60112445085c7d62a9061c0e0 2023-04-19 07:56:14,685 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/beb44b2152cf40579c71875b282b5750 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/beb44b2152cf40579c71875b282b5750 2023-04-19 07:56:14,689 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/beb44b2152cf40579c71875b282b5750, entries=150, sequenceid=783, filesize=11.9 K 2023-04-19 07:56:14,690 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/627039e1991f45038d28096a018bfabf as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/627039e1991f45038d28096a018bfabf 2023-04-19 07:56:14,695 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/627039e1991f45038d28096a018bfabf, entries=150, sequenceid=783, filesize=11.9 K 2023-04-19 07:56:14,695 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/0b9152b60112445085c7d62a9061c0e0 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0b9152b60112445085c7d62a9061c0e0 2023-04-19 07:56:14,699 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0b9152b60112445085c7d62a9061c0e0, entries=150, sequenceid=783, filesize=11.9 K 2023-04-19 07:56:14,700 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~428.95 KB/439248, currentSize=40.25 KB/41220 for 199c6b7f1f3a9353227e2a51c7965b15 in 1260ms, sequenceid=783, compaction requested=false 2023-04-19 07:56:14,700 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:14,700 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:14,701 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:14,701 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,701 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:14,701 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:14,702 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:14,702 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:14,703 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:14,703 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:14,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:14,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c2d2857[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:14,704 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:14,704 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:14,706 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:14,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:14,706 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:14,706 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:14,706 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:14,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:14,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:14,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:14,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:14,708 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:14,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:14,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:14,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:14,712 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:14,712 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:14,712 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:14,712 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:14,712 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:14,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:14,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:14,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:14,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:14,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:14,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:14,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:14,714 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:14,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:14,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:14,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:14,715 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:14,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:14,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:14,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:14,718 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:14,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:14,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:14,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:14,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:14,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:14,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:14,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:14,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:14,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:14,718 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:14,718 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:14,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:14,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:14,718 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:14,819 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:14,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:14,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:14,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:14,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:14,823 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:14,823 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:14,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:14,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:14,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:14,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:14,831 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,831 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:14,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:14,831 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:14,831 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:14,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:14,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:14,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:14,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:14,833 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:14,833 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:14,833 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:14,833 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 07:56:14,834 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:14,834 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:14,834 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:14,834 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:14,834 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:14,834 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:14,846 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=793 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/17164d5921e8465794db99e3eb0d1d7b 2023-04-19 07:56:15,266 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=793 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/5e8fa3a18c314d38a4362a0e276d295c 2023-04-19 07:56:15,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:15,614 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:15,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681890975631, max timestamp=1681890975651Num uniques 859; 2023-04-19 07:56:15,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:56:15,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681890975631, max timestamp=1681890975651Num uniques 859; 2023-04-19 07:56:15,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681890975631, max timestamp=1681890975651Num uniques 859; 2023-04-19 07:56:15,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:56:15,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-19 07:56:15,678 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=793 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/88c981ed0d9c475a9524ddfbf39b68ce 2023-04-19 07:56:15,682 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/17164d5921e8465794db99e3eb0d1d7b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/17164d5921e8465794db99e3eb0d1d7b 2023-04-19 07:56:15,685 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/17164d5921e8465794db99e3eb0d1d7b, entries=150, sequenceid=793, filesize=11.9 K 2023-04-19 07:56:15,686 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/5e8fa3a18c314d38a4362a0e276d295c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/5e8fa3a18c314d38a4362a0e276d295c 2023-04-19 07:56:15,688 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/5e8fa3a18c314d38a4362a0e276d295c, entries=150, sequenceid=793, filesize=11.9 K 2023-04-19 07:56:15,689 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/88c981ed0d9c475a9524ddfbf39b68ce as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/88c981ed0d9c475a9524ddfbf39b68ce 2023-04-19 07:56:15,692 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/88c981ed0d9c475a9524ddfbf39b68ce, entries=150, sequenceid=793, filesize=11.9 K 2023-04-19 07:56:15,692 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 199c6b7f1f3a9353227e2a51c7965b15 in 859ms, sequenceid=793, compaction requested=true 2023-04-19 07:56:15,693 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:15,693 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:15,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:15,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:15,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:15,693 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=154.31 KB heapSize=436.08 KB 2023-04-19 07:56:15,694 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:15,694 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:15,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:15,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 07:56:15,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681890975672, max timestamp=1681890975692Num uniques 1118; 2023-04-19 07:56:15,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-19 07:56:15,694 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:15,694 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:15,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:15,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 07:56:15,695 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:15,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:15,695 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:15,696 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,696 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:15,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:15,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:15,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a87e1fb[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:15,697 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:15,697 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,699 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,699 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:15,699 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:15,699 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:15,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:15,700 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:15,701 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:15,701 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:15,702 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/e08d5648ce9e49428903044d3f8d9efc 2023-04-19 07:56:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:15,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:15,703 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:15,704 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:15,704 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:15,704 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:15,704 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:15,707 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,707 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,707 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:15,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:15,708 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:15,709 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:15,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:15,712 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:15,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:15,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:15,712 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/5e45c9bc3ed44f03b3972ad01d811b29 2023-04-19 07:56:15,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:15,712 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:15,712 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:15,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:15,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:15,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,713 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,712 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:15,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:15,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:15,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:15,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891035713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:15,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891035715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:15,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891035716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,726 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:15,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891035717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,813 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:15,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:15,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:15,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:15,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:15,816 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:15,816 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:15,816 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:15,816 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:15,816 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,816 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:15,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:15,818 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:15,818 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,818 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:15,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:15,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:15,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:15,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:15,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:15,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:15,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:15,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:15,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:15,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:15,819 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:15,819 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:15,819 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:15,819 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:15,820 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:15,826 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,826 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:15,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:15,827 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:15,827 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:15,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:15,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ff794a2[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:15,829 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:15,829 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,833 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:15,833 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:15,833 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,833 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:15,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:15,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891035825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:15,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891035825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,835 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:15,835 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:15,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:15,835 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:15,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891035827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891035827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,835 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:15,837 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:15,838 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:15,838 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:15,838 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:15,838 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:15,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:15,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,840 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:15,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:15,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,841 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:15,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,842 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:15,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,844 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:15,844 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,844 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:15,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:15,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:15,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:15,844 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:15,844 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:15,844 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:15,844 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:15,844 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:15,845 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,845 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,845 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,845 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:15,945 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:15,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:15,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:15,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:15,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:15,948 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:15,948 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:15,948 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:15,948 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:15,948 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,948 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:15,949 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,949 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:15,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:15,950 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,950 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:15,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:15,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:15,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:15,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:15,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:15,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:15,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:15,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:15,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:15,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:15,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:15,951 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:15,951 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:15,951 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:15,951 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:15,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:15,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:15,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:15,951 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:15,952 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,953 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:15,953 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,953 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:15,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:15,954 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:15,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5665e9a1[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:15,954 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,957 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:15,957 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:15,957 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:15,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:15,958 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,958 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:15,958 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:15,958 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:15,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:15,960 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:15,960 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:15,960 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:15,960 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:15,960 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:15,962 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,962 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:15,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,962 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:15,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:15,963 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:15,964 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,966 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:15,966 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,966 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:15,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:15,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:15,966 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:15,966 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:15,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:15,967 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:15,967 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:15,967 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:15,967 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:15,967 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,967 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:15,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:15,967 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:16,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:16,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37896 deadline: 1681891036036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:16,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37928 deadline: 1681891036036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:16,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37878 deadline: 1681891036036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:16,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37912 deadline: 1681891036037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,067 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:16,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:16,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:16,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:16,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:16,070 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:16,070 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:16,070 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:16,070 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:16,071 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,071 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:16,073 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,073 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:16,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:16,073 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,073 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:16,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:16,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:16,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:16,074 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:16,074 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:16,074 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:16,074 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:16,074 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:16,074 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:16,074 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:16,075 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:16,075 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:16,075 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:16,076 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,076 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,077 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:16,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:16,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@439f1caa[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:16,078 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:16,078 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,079 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,079 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:16,079 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:16,079 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:16,079 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,080 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:16,081 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:16,081 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:16,081 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:16,081 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:16,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:16,083 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:16,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:16,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:16,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:16,083 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:16,085 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,085 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,085 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:16,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:16,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:16,086 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:16,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:16,087 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,091 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:16,091 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:16,091 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:16,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:16,092 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:16,092 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:16,092 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,092 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,092 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,092 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:16,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:16,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:16,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:16,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:16,092 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:16,092 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:16,121 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/55cd404373234b0487393d6042e6edc1 2023-04-19 07:56:16,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/e08d5648ce9e49428903044d3f8d9efc as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/e08d5648ce9e49428903044d3f8d9efc 2023-04-19 07:56:16,128 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/e08d5648ce9e49428903044d3f8d9efc, entries=300, sequenceid=832, filesize=19.0 K 2023-04-19 07:56:16,129 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/5e45c9bc3ed44f03b3972ad01d811b29 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/5e45c9bc3ed44f03b3972ad01d811b29 2023-04-19 07:56:16,134 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/5e45c9bc3ed44f03b3972ad01d811b29, entries=150, sequenceid=832, filesize=11.9 K 2023-04-19 07:56:16,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/55cd404373234b0487393d6042e6edc1 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/55cd404373234b0487393d6042e6edc1 2023-04-19 07:56:16,139 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/55cd404373234b0487393d6042e6edc1, entries=150, sequenceid=832, filesize=11.9 K 2023-04-19 07:56:16,140 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~337.47 KB/345568, currentSize=60.38 KB/61830 for 199c6b7f1f3a9353227e2a51c7965b15 in 447ms, sequenceid=832, compaction requested=true 2023-04-19 07:56:16,140 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:16,140 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:56:16,140 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:16,141 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:56:16,141 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:16,141 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:16,142 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 57120 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:56:16,142 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/A is initiating minor compaction (all files) 2023-04-19 07:56:16,142 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/A in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:16,142 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/ce40183fd04e45cdb82e2e129d34d6a7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/beb44b2152cf40579c71875b282b5750, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/17164d5921e8465794db99e3eb0d1d7b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/e08d5648ce9e49428903044d3f8d9efc] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=55.8 K 2023-04-19 07:56:16,143 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:56:16,143 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/B is initiating minor compaction (all files) 2023-04-19 07:56:16,143 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/B in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:16,143 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/f690ccbb7cb34ecf8345fa7b7f62e645, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/627039e1991f45038d28096a018bfabf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/5e8fa3a18c314d38a4362a0e276d295c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/5e45c9bc3ed44f03b3972ad01d811b29] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=48.6 K 2023-04-19 07:56:16,144 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting f690ccbb7cb34ecf8345fa7b7f62e645, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1681890972421 2023-04-19 07:56:16,144 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 627039e1991f45038d28096a018bfabf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1681890973146 2023-04-19 07:56:16,144 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting ce40183fd04e45cdb82e2e129d34d6a7, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1681890972421 2023-04-19 07:56:16,144 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 5e8fa3a18c314d38a4362a0e276d295c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=793, earliestPutTs=1681890973473 2023-04-19 07:56:16,145 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting beb44b2152cf40579c71875b282b5750, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1681890973146 2023-04-19 07:56:16,145 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 5e45c9bc3ed44f03b3972ad01d811b29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1681890975682 2023-04-19 07:56:16,146 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 17164d5921e8465794db99e3eb0d1d7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=793, earliestPutTs=1681890973473 2023-04-19 07:56:16,147 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting e08d5648ce9e49428903044d3f8d9efc, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1681890975672 2023-04-19 07:56:16,158 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#B#compaction#489 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:16,162 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#A#compaction#490 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:16,187 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/d1fbde5dd7b74031a119e0c016c45973 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/d1fbde5dd7b74031a119e0c016c45973 2023-04-19 07:56:16,190 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/7532ae5ce4ab41aa8031a10ab082237f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/7532ae5ce4ab41aa8031a10ab082237f 2023-04-19 07:56:16,192 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:16,193 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/B of 199c6b7f1f3a9353227e2a51c7965b15 into d1fbde5dd7b74031a119e0c016c45973(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:16,193 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:16,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:16,193 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/B, priority=12, startTime=1681890976141; duration=0sec 2023-04-19 07:56:16,195 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:16,195 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:56:16,195 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/A of 199c6b7f1f3a9353227e2a51c7965b15 into 7532ae5ce4ab41aa8031a10ab082237f(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:16,195 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:16,195 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/A, priority=12, startTime=1681890976140; duration=0sec 2023-04-19 07:56:16,195 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:16,196 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:56:16,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:16,197 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 199c6b7f1f3a9353227e2a51c7965b15/C is initiating minor compaction (all files) 2023-04-19 07:56:16,197 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 199c6b7f1f3a9353227e2a51c7965b15/C in TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:16,197 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/4590f1e1ffa443deaa3a589ca92dccf0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0b9152b60112445085c7d62a9061c0e0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/88c981ed0d9c475a9524ddfbf39b68ce, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/55cd404373234b0487393d6042e6edc1] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp, totalSize=48.6 K 2023-04-19 07:56:16,197 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 4590f1e1ffa443deaa3a589ca92dccf0, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1681890972421 2023-04-19 07:56:16,198 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 0b9152b60112445085c7d62a9061c0e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=783, earliestPutTs=1681890973146 2023-04-19 07:56:16,198 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 88c981ed0d9c475a9524ddfbf39b68ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=793, earliestPutTs=1681890973473 2023-04-19 07:56:16,199 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 55cd404373234b0487393d6042e6edc1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1681890975682 2023-04-19 07:56:16,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:16,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:16,200 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:16,200 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:16,200 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:16,200 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:16,200 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,200 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:16,201 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:16,201 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:16,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:16,202 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,202 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:16,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:16,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:16,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:16,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:16,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:16,203 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:16,204 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:16,204 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. started... 2023-04-19 07:56:16,205 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-19 07:56:16,205 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:16,205 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:16,205 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:16,205 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:16,205 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:16,205 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:16,205 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 199c6b7f1f3a9353227e2a51c7965b15#C#compaction#491 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:16,233 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=846 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/847e4b5c48d64fd8b243b4594faba44c 2023-04-19 07:56:16,240 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=846 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/4125b1e237c74eb3a731d892800ebaa2 2023-04-19 07:56:16,248 DEBUG [Thread-2964] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5726fb8b to 127.0.0.1:58087 2023-04-19 07:56:16,248 DEBUG [Thread-2964] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:16,249 DEBUG [Thread-2966] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f2b3c03 to 127.0.0.1:58087 2023-04-19 07:56:16,249 DEBUG [Thread-2966] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:16,254 DEBUG [Thread-2972] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78592df0 to 127.0.0.1:58087 2023-04-19 07:56:16,254 DEBUG [Thread-2972] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:16,254 DEBUG [Thread-2968] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e2be04 to 127.0.0.1:58087 2023-04-19 07:56:16,254 DEBUG [Thread-2968] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:16,255 DEBUG [Thread-2970] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x067ea274 to 127.0.0.1:58087 2023-04-19 07:56:16,255 DEBUG [Thread-2970] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:16,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:16,341 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. as already flushing 2023-04-19 07:56:16,342 DEBUG [Thread-2957] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37705401 to 127.0.0.1:58087 2023-04-19 07:56:16,342 DEBUG [Thread-2961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x052203cb to 127.0.0.1:58087 2023-04-19 07:56:16,342 DEBUG [Thread-2957] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:16,342 DEBUG [Thread-2961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:16,342 DEBUG [Thread-2955] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a987907 to 127.0.0.1:58087 2023-04-19 07:56:16,342 DEBUG [Thread-2955] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:16,343 DEBUG [Thread-2953] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73eb7080 to 127.0.0.1:58087 2023-04-19 07:56:16,343 DEBUG [Thread-2953] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:16,629 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/86ca736d23184f6495acf129f233f7d1 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/86ca736d23184f6495acf129f233f7d1 2023-04-19 07:56:16,632 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 199c6b7f1f3a9353227e2a51c7965b15/C of 199c6b7f1f3a9353227e2a51c7965b15 into 86ca736d23184f6495acf129f233f7d1(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:16,633 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:16,633 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15., storeName=199c6b7f1f3a9353227e2a51c7965b15/C, priority=12, startTime=1681890976141; duration=0sec 2023-04-19 07:56:16,633 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:16,649 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=846 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/c2b1ea84a35643eebcc23ad91e78cf8b 2023-04-19 07:56:16,652 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/847e4b5c48d64fd8b243b4594faba44c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/847e4b5c48d64fd8b243b4594faba44c 2023-04-19 07:56:16,657 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/847e4b5c48d64fd8b243b4594faba44c, entries=150, sequenceid=846, filesize=11.9 K 2023-04-19 07:56:16,657 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/4125b1e237c74eb3a731d892800ebaa2 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/4125b1e237c74eb3a731d892800ebaa2 2023-04-19 07:56:16,660 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/4125b1e237c74eb3a731d892800ebaa2, entries=150, sequenceid=846, filesize=11.9 K 2023-04-19 07:56:16,661 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/c2b1ea84a35643eebcc23ad91e78cf8b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/c2b1ea84a35643eebcc23ad91e78cf8b 2023-04-19 07:56:16,665 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/c2b1ea84a35643eebcc23ad91e78cf8b, entries=150, sequenceid=846, filesize=11.9 K 2023-04-19 07:56:16,665 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=26.84 KB/27480 for 199c6b7f1f3a9353227e2a51c7965b15 in 460ms, sequenceid=846, compaction requested=false 2023-04-19 07:56:16,665 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:16,665 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:16,665 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:16,666 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:16,666 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,666 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:16,666 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:16,667 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,667 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:16,667 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,667 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:16,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:16,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:16,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2672b51b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:16,669 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:16,669 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,670 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:16,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:16,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:16,670 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,670 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:16,672 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:16,672 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:16,672 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,672 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:16,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:16,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:16,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:16,674 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:16,674 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:16,674 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:16,674 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:16,674 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:16,675 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,675 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:16,676 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:16,676 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:16,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,678 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,678 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:16,678 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:16,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:16,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:16,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:16,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:16,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:16,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:16,679 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:16,679 DEBUG [Thread-2963] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:16,679 DEBUG [Thread-2963] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:16,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:16,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:16,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:16,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:16,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:16,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:16,779 DEBUG [Thread-2963] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:16,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:20,724 DEBUG [Thread-2959] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15880ffb to 127.0.0.1:58087 2023-04-19 07:56:20,724 DEBUG [Thread-2959] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:20,724 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-19 07:56:20,724 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 196 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 195 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 160 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 134 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(402): scanned 1825 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(403): verified 5475 rows 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(402): scanned 1834 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(403): verified 5493 rows 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(402): scanned 1831 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(403): verified 5493 rows 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(402): scanned 1839 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(403): verified 5517 rows 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(402): scanned 1822 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(403): verified 5466 rows 2023-04-19 07:56:20,725 INFO [Listener at localhost/42517] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-19 07:56:20,725 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bec116b to 127.0.0.1:58087 2023-04-19 07:56:20,725 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:20,727 INFO [Listener at localhost/42517] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-19 07:56:20,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-19 07:56:20,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:20,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-19 07:56:20,732 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890980732"}]},"ts":"1681890980732"} 2023-04-19 07:56:20,734 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-19 07:56:20,829 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-19 07:56:20,830 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=199c6b7f1f3a9353227e2a51c7965b15, UNASSIGN}] 2023-04-19 07:56:20,831 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=199c6b7f1f3a9353227e2a51c7965b15, UNASSIGN 2023-04-19 07:56:20,831 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=199c6b7f1f3a9353227e2a51c7965b15, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:20,832 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890980831"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890980831"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890980831"}]},"ts":"1681890980831"} 2023-04-19 07:56:20,833 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:56:20,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-19 07:56:20,984 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:20,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 199c6b7f1f3a9353227e2a51c7965b15, disabling compactions & flushes 2023-04-19 07:56:20,984 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:20,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:20,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. after waiting 0 ms 2023-04-19 07:56:20,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:20,984 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 199c6b7f1f3a9353227e2a51c7965b15 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-19 07:56:20,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=A 2023-04-19 07:56:20,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:20,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=B 2023-04-19 07:56:20,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:20,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 199c6b7f1f3a9353227e2a51c7965b15, store=C 2023-04-19 07:56:20,985 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:21,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-19 07:56:21,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-19 07:56:21,391 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=855 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/911a617bc66b4ef081f0216cd5c45811 2023-04-19 07:56:21,801 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=855 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/f8d0d255f86e4329bf87eafabd38fb17 2023-04-19 07:56:21,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-19 07:56:22,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=855 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/b0d93c9eb136492daf212a1d350060c1 2023-04-19 07:56:22,212 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/A/911a617bc66b4ef081f0216cd5c45811 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/911a617bc66b4ef081f0216cd5c45811 2023-04-19 07:56:22,215 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/911a617bc66b4ef081f0216cd5c45811, entries=100, sequenceid=855, filesize=9.5 K 2023-04-19 07:56:22,216 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/B/f8d0d255f86e4329bf87eafabd38fb17 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/f8d0d255f86e4329bf87eafabd38fb17 2023-04-19 07:56:22,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/f8d0d255f86e4329bf87eafabd38fb17, entries=100, sequenceid=855, filesize=9.5 K 2023-04-19 07:56:22,219 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/.tmp/C/b0d93c9eb136492daf212a1d350060c1 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b0d93c9eb136492daf212a1d350060c1 2023-04-19 07:56:22,222 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b0d93c9eb136492daf212a1d350060c1, entries=100, sequenceid=855, filesize=9.5 K 2023-04-19 07:56:22,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 199c6b7f1f3a9353227e2a51c7965b15 in 1239ms, sequenceid=855, compaction requested=true 2023-04-19 07:56:22,224 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/c3bc5ba20cb94d43b5ebd34778713576, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/7b914d2a74ec439b9e4051e34661e8f9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/0d1d458d450749e59ff0ef607f008a44, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/619f738c707b4adfb5b99a63e3bb2348, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/3ee656e5b5364abeb7d242c7c612fda5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/4879c16199354307a3f446b0c7a124d4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/32fc60d3b49f4f09b10b7e136a87a6f7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/68ad19c82a64478c83878319e145d17f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/914b73a88b1f436eacd250184ab831af, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/0f73bfb47cbf4171ba23dbb3e16f9a43, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/aa436fda9d074f08af62de80f17fb6df, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d4a9ee4f3d42472cbfbb79a9ee9d209a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/80b32d871a2f4ff7a08a1acc5e09bd1d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/794d2e6c7b6f43c492fb2996f7869c66, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d6357e0f085f4b1aa6035792391198bf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/f090e36c516543df99081c5a9528157a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/7fdac7e3e3eb4ad59df8eb5a795c0721, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/5d7bbd32126940e99c2ed1f2e465244b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/8f59454e67e941ddaa4193d2ab33f069, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/20a0fd4d32584c8984ce138d18b85dc9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/71ef49dea68448e2b7301533e5fd3079, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/3c411b78a9b74d75b251ebd85714d54d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/b535673690244e0ea0ee065441be78b7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/bace4c8adecc4ab6a20be6d096e20192, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/e40f0880d1634c91bcbcded9c849e213, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/fd0297c71067406ab4ed7d74ff59f8e8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/6692ec19816b41b88638ae866c21d686, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/f0ad67f4654c41318a8ec656228de284, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d8900e45edb24209a98b29eb51bc94af, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/fc1a0eded4114d1a9427f4a2205c30ff, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/c1a9ff30ece24e02a3624496c004d96c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/ce40183fd04e45cdb82e2e129d34d6a7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/beb44b2152cf40579c71875b282b5750, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/17164d5921e8465794db99e3eb0d1d7b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/e08d5648ce9e49428903044d3f8d9efc] to archive 2023-04-19 07:56:22,225 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:56:22,227 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/c3bc5ba20cb94d43b5ebd34778713576 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/c3bc5ba20cb94d43b5ebd34778713576 2023-04-19 07:56:22,228 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/7b914d2a74ec439b9e4051e34661e8f9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/7b914d2a74ec439b9e4051e34661e8f9 2023-04-19 07:56:22,229 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/0d1d458d450749e59ff0ef607f008a44 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/0d1d458d450749e59ff0ef607f008a44 2023-04-19 07:56:22,230 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/619f738c707b4adfb5b99a63e3bb2348 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/619f738c707b4adfb5b99a63e3bb2348 2023-04-19 07:56:22,230 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/3ee656e5b5364abeb7d242c7c612fda5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/3ee656e5b5364abeb7d242c7c612fda5 2023-04-19 07:56:22,231 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/4879c16199354307a3f446b0c7a124d4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/4879c16199354307a3f446b0c7a124d4 2023-04-19 07:56:22,232 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/32fc60d3b49f4f09b10b7e136a87a6f7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/32fc60d3b49f4f09b10b7e136a87a6f7 2023-04-19 07:56:22,233 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/68ad19c82a64478c83878319e145d17f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/68ad19c82a64478c83878319e145d17f 2023-04-19 07:56:22,233 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/914b73a88b1f436eacd250184ab831af to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/914b73a88b1f436eacd250184ab831af 2023-04-19 07:56:22,234 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/0f73bfb47cbf4171ba23dbb3e16f9a43 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/0f73bfb47cbf4171ba23dbb3e16f9a43 2023-04-19 07:56:22,235 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/aa436fda9d074f08af62de80f17fb6df to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/aa436fda9d074f08af62de80f17fb6df 2023-04-19 07:56:22,236 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d4a9ee4f3d42472cbfbb79a9ee9d209a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d4a9ee4f3d42472cbfbb79a9ee9d209a 2023-04-19 07:56:22,237 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/80b32d871a2f4ff7a08a1acc5e09bd1d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/80b32d871a2f4ff7a08a1acc5e09bd1d 2023-04-19 07:56:22,237 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/794d2e6c7b6f43c492fb2996f7869c66 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/794d2e6c7b6f43c492fb2996f7869c66 2023-04-19 07:56:22,238 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d6357e0f085f4b1aa6035792391198bf to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d6357e0f085f4b1aa6035792391198bf 2023-04-19 07:56:22,239 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/f090e36c516543df99081c5a9528157a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/f090e36c516543df99081c5a9528157a 2023-04-19 07:56:22,239 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/7fdac7e3e3eb4ad59df8eb5a795c0721 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/7fdac7e3e3eb4ad59df8eb5a795c0721 2023-04-19 07:56:22,240 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/5d7bbd32126940e99c2ed1f2e465244b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/5d7bbd32126940e99c2ed1f2e465244b 2023-04-19 07:56:22,241 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/8f59454e67e941ddaa4193d2ab33f069 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/8f59454e67e941ddaa4193d2ab33f069 2023-04-19 07:56:22,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/20a0fd4d32584c8984ce138d18b85dc9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/20a0fd4d32584c8984ce138d18b85dc9 2023-04-19 07:56:22,242 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/71ef49dea68448e2b7301533e5fd3079 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/71ef49dea68448e2b7301533e5fd3079 2023-04-19 07:56:22,243 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/3c411b78a9b74d75b251ebd85714d54d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/3c411b78a9b74d75b251ebd85714d54d 2023-04-19 07:56:22,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/b535673690244e0ea0ee065441be78b7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/b535673690244e0ea0ee065441be78b7 2023-04-19 07:56:22,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/bace4c8adecc4ab6a20be6d096e20192 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/bace4c8adecc4ab6a20be6d096e20192 2023-04-19 07:56:22,245 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/e40f0880d1634c91bcbcded9c849e213 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/e40f0880d1634c91bcbcded9c849e213 2023-04-19 07:56:22,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/fd0297c71067406ab4ed7d74ff59f8e8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/fd0297c71067406ab4ed7d74ff59f8e8 2023-04-19 07:56:22,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/6692ec19816b41b88638ae866c21d686 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/6692ec19816b41b88638ae866c21d686 2023-04-19 07:56:22,247 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/f0ad67f4654c41318a8ec656228de284 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/f0ad67f4654c41318a8ec656228de284 2023-04-19 07:56:22,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d8900e45edb24209a98b29eb51bc94af to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/d8900e45edb24209a98b29eb51bc94af 2023-04-19 07:56:22,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/fc1a0eded4114d1a9427f4a2205c30ff to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/fc1a0eded4114d1a9427f4a2205c30ff 2023-04-19 07:56:22,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/c1a9ff30ece24e02a3624496c004d96c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/c1a9ff30ece24e02a3624496c004d96c 2023-04-19 07:56:22,250 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/ce40183fd04e45cdb82e2e129d34d6a7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/ce40183fd04e45cdb82e2e129d34d6a7 2023-04-19 07:56:22,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/beb44b2152cf40579c71875b282b5750 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/beb44b2152cf40579c71875b282b5750 2023-04-19 07:56:22,252 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/17164d5921e8465794db99e3eb0d1d7b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/17164d5921e8465794db99e3eb0d1d7b 2023-04-19 07:56:22,253 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/e08d5648ce9e49428903044d3f8d9efc to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/e08d5648ce9e49428903044d3f8d9efc 2023-04-19 07:56:22,256 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/04e08a792716461f8eea445466a58d44, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/53d11f47046646c5990cc5edbf9f2bb8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/87d68a94653b4b67b31b8c25fa776769, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/6372678b64a14a0694dbb6451f67a48e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/3ae3f7b72852401e9ef7d1030e33bb23, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2e097fa8064e4f92b645489d17538b28, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/69357012cc4c424aaa6cec4332778d58, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/acad54861c0d4f21b8c4189f91330dcc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/3a47a84c53d94ded89fab4480be777d8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/811ccea87c6b4d82863c359016df2857, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/93d6f1d5d857462da99b3e75c2e933d0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/80b24338fc3c4c7d82c92152c17fea5f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/19c523d8f7994f73b28afe1574124e43, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/c3e589b42de443d39091468667f60938, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/31569ca5d0ce4f8c8237546d7d27ad04, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/f51d8afd3f234cebad4fdb4a4f3672ab, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/91a84f4b709040e1886367a97f9216f6, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/1ea82af70386489188115e8a142465c5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/160a901bcb864794899e3278d7abbbd3, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/b987c1c8316f4a4b8beeae4fb0ab8209, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2d3b1696ef6a48fb98076c6c0210a27d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/caf743df133e4dd2bc56b7cb9d6710b5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/46ff68515b7f4c0d8075db6600e61e5b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2516cd48867344c385b9b5d56e160899, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/7d9cc1641ef24763b5401fe3007ab9c4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/bf676ce9b6c54048848d8fe08e977606, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/0ab8713a06c047f49835d5a41a211b8a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/7a22bd66598f437b96599a6db54ed4d8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/fb4f55c9fea14674b5a3e5eb2e9753a1, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/502fb839661e4bb8b35739211a7f1045, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/f690ccbb7cb34ecf8345fa7b7f62e645, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/53f49b4bdbaa44e387d9e08ef27137ca, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/627039e1991f45038d28096a018bfabf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/5e8fa3a18c314d38a4362a0e276d295c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/5e45c9bc3ed44f03b3972ad01d811b29] to archive 2023-04-19 07:56:22,256 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:56:22,258 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/04e08a792716461f8eea445466a58d44 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/04e08a792716461f8eea445466a58d44 2023-04-19 07:56:22,259 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/53d11f47046646c5990cc5edbf9f2bb8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/53d11f47046646c5990cc5edbf9f2bb8 2023-04-19 07:56:22,259 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/87d68a94653b4b67b31b8c25fa776769 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/87d68a94653b4b67b31b8c25fa776769 2023-04-19 07:56:22,260 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/6372678b64a14a0694dbb6451f67a48e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/6372678b64a14a0694dbb6451f67a48e 2023-04-19 07:56:22,261 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/3ae3f7b72852401e9ef7d1030e33bb23 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/3ae3f7b72852401e9ef7d1030e33bb23 2023-04-19 07:56:22,262 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2e097fa8064e4f92b645489d17538b28 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2e097fa8064e4f92b645489d17538b28 2023-04-19 07:56:22,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/69357012cc4c424aaa6cec4332778d58 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/69357012cc4c424aaa6cec4332778d58 2023-04-19 07:56:22,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/acad54861c0d4f21b8c4189f91330dcc to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/acad54861c0d4f21b8c4189f91330dcc 2023-04-19 07:56:22,264 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/3a47a84c53d94ded89fab4480be777d8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/3a47a84c53d94ded89fab4480be777d8 2023-04-19 07:56:22,265 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/811ccea87c6b4d82863c359016df2857 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/811ccea87c6b4d82863c359016df2857 2023-04-19 07:56:22,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/93d6f1d5d857462da99b3e75c2e933d0 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/93d6f1d5d857462da99b3e75c2e933d0 2023-04-19 07:56:22,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/80b24338fc3c4c7d82c92152c17fea5f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/80b24338fc3c4c7d82c92152c17fea5f 2023-04-19 07:56:22,267 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/19c523d8f7994f73b28afe1574124e43 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/19c523d8f7994f73b28afe1574124e43 2023-04-19 07:56:22,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/c3e589b42de443d39091468667f60938 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/c3e589b42de443d39091468667f60938 2023-04-19 07:56:22,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/31569ca5d0ce4f8c8237546d7d27ad04 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/31569ca5d0ce4f8c8237546d7d27ad04 2023-04-19 07:56:22,269 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/f51d8afd3f234cebad4fdb4a4f3672ab to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/f51d8afd3f234cebad4fdb4a4f3672ab 2023-04-19 07:56:22,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/91a84f4b709040e1886367a97f9216f6 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/91a84f4b709040e1886367a97f9216f6 2023-04-19 07:56:22,271 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/1ea82af70386489188115e8a142465c5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/1ea82af70386489188115e8a142465c5 2023-04-19 07:56:22,272 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/160a901bcb864794899e3278d7abbbd3 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/160a901bcb864794899e3278d7abbbd3 2023-04-19 07:56:22,272 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/b987c1c8316f4a4b8beeae4fb0ab8209 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/b987c1c8316f4a4b8beeae4fb0ab8209 2023-04-19 07:56:22,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2d3b1696ef6a48fb98076c6c0210a27d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2d3b1696ef6a48fb98076c6c0210a27d 2023-04-19 07:56:22,274 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/caf743df133e4dd2bc56b7cb9d6710b5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/caf743df133e4dd2bc56b7cb9d6710b5 2023-04-19 07:56:22,275 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/46ff68515b7f4c0d8075db6600e61e5b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/46ff68515b7f4c0d8075db6600e61e5b 2023-04-19 07:56:22,276 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2516cd48867344c385b9b5d56e160899 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/2516cd48867344c385b9b5d56e160899 2023-04-19 07:56:22,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/7d9cc1641ef24763b5401fe3007ab9c4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/7d9cc1641ef24763b5401fe3007ab9c4 2023-04-19 07:56:22,278 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/bf676ce9b6c54048848d8fe08e977606 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/bf676ce9b6c54048848d8fe08e977606 2023-04-19 07:56:22,279 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/0ab8713a06c047f49835d5a41a211b8a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/0ab8713a06c047f49835d5a41a211b8a 2023-04-19 07:56:22,280 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/7a22bd66598f437b96599a6db54ed4d8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/7a22bd66598f437b96599a6db54ed4d8 2023-04-19 07:56:22,281 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/fb4f55c9fea14674b5a3e5eb2e9753a1 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/fb4f55c9fea14674b5a3e5eb2e9753a1 2023-04-19 07:56:22,282 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/502fb839661e4bb8b35739211a7f1045 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/502fb839661e4bb8b35739211a7f1045 2023-04-19 07:56:22,283 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/f690ccbb7cb34ecf8345fa7b7f62e645 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/f690ccbb7cb34ecf8345fa7b7f62e645 2023-04-19 07:56:22,284 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/53f49b4bdbaa44e387d9e08ef27137ca to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/53f49b4bdbaa44e387d9e08ef27137ca 2023-04-19 07:56:22,285 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/627039e1991f45038d28096a018bfabf to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/627039e1991f45038d28096a018bfabf 2023-04-19 07:56:22,286 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/5e8fa3a18c314d38a4362a0e276d295c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/5e8fa3a18c314d38a4362a0e276d295c 2023-04-19 07:56:22,287 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/5e45c9bc3ed44f03b3972ad01d811b29 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/5e45c9bc3ed44f03b3972ad01d811b29 2023-04-19 07:56:22,291 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/c2028394db71437f8b778ae91de25013, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/db6dcea247c94f0bb56efd243990c0fa, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/f1ffa481ad2d496fa23003dc06e2079d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/bd0bfecd35914c0597971f220325d632, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/7200999589db4cf5a62dc439cf0df2aa, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0ad3ad7b58f74506ba485a9444782f5c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/3dd76cce922945188e8b037f3800447f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/304e7b57d3244f4c881be1d68c3c378a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b10e1361dd9b4b36a269c6079ff9affa, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/8228f57684dc4dc3bd048b89447cb159, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/bb770ec1981e403289fa9cbd0c7b51a4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b035a25347304d00b23c7006648814f8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a827c93349a5499a8d58d4994b2652dd, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/ac599d546e8342ec8d7c8fffa49d9a31, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/ce14e03e2287471b976c21ad347e80c2, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/9e7f4ac311544aaa9998c7f96a86e9b5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/42750a62293f4e86a92ed1d79b0cbcae, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a1f6d12bd18348bb8f5bdfd045062702, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/f6ba06c380a4426fa5733906aeaa746d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/6ead41732a8249ef9767e47d083026b4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/478136419d87497f84c67f3cfcdf70ac, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/5b202ca9b6d14e0a81b29f007e7db001, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/d4f911fb532b4fe19213914f1bc5ea38, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/7a114ccddc474f37b3768c6916ecaa05, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a0d623c5c960455aa16fc5d64753eb0b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/be10b73006914c5297b62f691e699a83, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0e9ab4bca2fd46b9a2b83eea00369b52, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/1f29e0f26c484f5ab3f8b4e5d8022735, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/5c97266aefb84ea8bd5263fcf5013fa0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/6f985faca6fa4943b77f0ddc0a0e9c34, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/4590f1e1ffa443deaa3a589ca92dccf0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/77347b6a4f144e6f8401a03da8847246, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0b9152b60112445085c7d62a9061c0e0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/88c981ed0d9c475a9524ddfbf39b68ce, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/55cd404373234b0487393d6042e6edc1] to archive 2023-04-19 07:56:22,291 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:56:22,293 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/c2028394db71437f8b778ae91de25013 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/c2028394db71437f8b778ae91de25013 2023-04-19 07:56:22,294 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/db6dcea247c94f0bb56efd243990c0fa to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/db6dcea247c94f0bb56efd243990c0fa 2023-04-19 07:56:22,295 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/f1ffa481ad2d496fa23003dc06e2079d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/f1ffa481ad2d496fa23003dc06e2079d 2023-04-19 07:56:22,296 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/bd0bfecd35914c0597971f220325d632 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/bd0bfecd35914c0597971f220325d632 2023-04-19 07:56:22,299 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/7200999589db4cf5a62dc439cf0df2aa to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/7200999589db4cf5a62dc439cf0df2aa 2023-04-19 07:56:22,300 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0ad3ad7b58f74506ba485a9444782f5c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0ad3ad7b58f74506ba485a9444782f5c 2023-04-19 07:56:22,301 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/3dd76cce922945188e8b037f3800447f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/3dd76cce922945188e8b037f3800447f 2023-04-19 07:56:22,302 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/304e7b57d3244f4c881be1d68c3c378a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/304e7b57d3244f4c881be1d68c3c378a 2023-04-19 07:56:22,303 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b10e1361dd9b4b36a269c6079ff9affa to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b10e1361dd9b4b36a269c6079ff9affa 2023-04-19 07:56:22,304 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/8228f57684dc4dc3bd048b89447cb159 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/8228f57684dc4dc3bd048b89447cb159 2023-04-19 07:56:22,305 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/bb770ec1981e403289fa9cbd0c7b51a4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/bb770ec1981e403289fa9cbd0c7b51a4 2023-04-19 07:56:22,306 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b035a25347304d00b23c7006648814f8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b035a25347304d00b23c7006648814f8 2023-04-19 07:56:22,307 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a827c93349a5499a8d58d4994b2652dd to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a827c93349a5499a8d58d4994b2652dd 2023-04-19 07:56:22,308 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/ac599d546e8342ec8d7c8fffa49d9a31 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/ac599d546e8342ec8d7c8fffa49d9a31 2023-04-19 07:56:22,309 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/ce14e03e2287471b976c21ad347e80c2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/ce14e03e2287471b976c21ad347e80c2 2023-04-19 07:56:22,309 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/9e7f4ac311544aaa9998c7f96a86e9b5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/9e7f4ac311544aaa9998c7f96a86e9b5 2023-04-19 07:56:22,310 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/42750a62293f4e86a92ed1d79b0cbcae to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/42750a62293f4e86a92ed1d79b0cbcae 2023-04-19 07:56:22,311 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a1f6d12bd18348bb8f5bdfd045062702 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a1f6d12bd18348bb8f5bdfd045062702 2023-04-19 07:56:22,312 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/f6ba06c380a4426fa5733906aeaa746d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/f6ba06c380a4426fa5733906aeaa746d 2023-04-19 07:56:22,313 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/6ead41732a8249ef9767e47d083026b4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/6ead41732a8249ef9767e47d083026b4 2023-04-19 07:56:22,313 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/478136419d87497f84c67f3cfcdf70ac to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/478136419d87497f84c67f3cfcdf70ac 2023-04-19 07:56:22,314 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/5b202ca9b6d14e0a81b29f007e7db001 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/5b202ca9b6d14e0a81b29f007e7db001 2023-04-19 07:56:22,315 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/d4f911fb532b4fe19213914f1bc5ea38 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/d4f911fb532b4fe19213914f1bc5ea38 2023-04-19 07:56:22,316 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/7a114ccddc474f37b3768c6916ecaa05 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/7a114ccddc474f37b3768c6916ecaa05 2023-04-19 07:56:22,317 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a0d623c5c960455aa16fc5d64753eb0b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/a0d623c5c960455aa16fc5d64753eb0b 2023-04-19 07:56:22,317 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/be10b73006914c5297b62f691e699a83 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/be10b73006914c5297b62f691e699a83 2023-04-19 07:56:22,318 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0e9ab4bca2fd46b9a2b83eea00369b52 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0e9ab4bca2fd46b9a2b83eea00369b52 2023-04-19 07:56:22,319 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/1f29e0f26c484f5ab3f8b4e5d8022735 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/1f29e0f26c484f5ab3f8b4e5d8022735 2023-04-19 07:56:22,320 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/5c97266aefb84ea8bd5263fcf5013fa0 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/5c97266aefb84ea8bd5263fcf5013fa0 2023-04-19 07:56:22,320 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/6f985faca6fa4943b77f0ddc0a0e9c34 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/6f985faca6fa4943b77f0ddc0a0e9c34 2023-04-19 07:56:22,321 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/4590f1e1ffa443deaa3a589ca92dccf0 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/4590f1e1ffa443deaa3a589ca92dccf0 2023-04-19 07:56:22,322 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/77347b6a4f144e6f8401a03da8847246 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/77347b6a4f144e6f8401a03da8847246 2023-04-19 07:56:22,323 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0b9152b60112445085c7d62a9061c0e0 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/0b9152b60112445085c7d62a9061c0e0 2023-04-19 07:56:22,323 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/88c981ed0d9c475a9524ddfbf39b68ce to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/88c981ed0d9c475a9524ddfbf39b68ce 2023-04-19 07:56:22,324 DEBUG [StoreCloser-TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/55cd404373234b0487393d6042e6edc1 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/55cd404373234b0487393d6042e6edc1 2023-04-19 07:56:22,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/recovered.edits/858.seqid, newMaxSeqId=858, maxSeqId=1 2023-04-19 07:56:22,329 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15. 2023-04-19 07:56:22,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 199c6b7f1f3a9353227e2a51c7965b15: 2023-04-19 07:56:22,331 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:22,331 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=199c6b7f1f3a9353227e2a51c7965b15, regionState=CLOSED 2023-04-19 07:56:22,331 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890982331"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890982331"}]},"ts":"1681890982331"} 2023-04-19 07:56:22,333 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-19 07:56:22,333 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 199c6b7f1f3a9353227e2a51c7965b15, server=jenkins-hbase4.apache.org,42359,1681890819191 in 1.4990 sec 2023-04-19 07:56:22,334 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-19 07:56:22,334 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=199c6b7f1f3a9353227e2a51c7965b15, UNASSIGN in 1.5030 sec 2023-04-19 07:56:22,335 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890982334"}]},"ts":"1681890982334"} 2023-04-19 07:56:22,335 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-19 07:56:22,337 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-19 07:56:22,338 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6100 sec 2023-04-19 07:56:22,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-19 07:56:22,837 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-19 07:56:22,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-19 07:56:22,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:22,839 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:22,840 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:22,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-19 07:56:22,844 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:22,846 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/recovered.edits] 2023-04-19 07:56:22,848 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/7532ae5ce4ab41aa8031a10ab082237f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/7532ae5ce4ab41aa8031a10ab082237f 2023-04-19 07:56:22,849 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/847e4b5c48d64fd8b243b4594faba44c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/847e4b5c48d64fd8b243b4594faba44c 2023-04-19 07:56:22,850 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/911a617bc66b4ef081f0216cd5c45811 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/A/911a617bc66b4ef081f0216cd5c45811 2023-04-19 07:56:22,851 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/4125b1e237c74eb3a731d892800ebaa2 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/4125b1e237c74eb3a731d892800ebaa2 2023-04-19 07:56:22,852 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/d1fbde5dd7b74031a119e0c016c45973 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/d1fbde5dd7b74031a119e0c016c45973 2023-04-19 07:56:22,853 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/f8d0d255f86e4329bf87eafabd38fb17 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/B/f8d0d255f86e4329bf87eafabd38fb17 2023-04-19 07:56:22,855 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/86ca736d23184f6495acf129f233f7d1 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/86ca736d23184f6495acf129f233f7d1 2023-04-19 07:56:22,856 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b0d93c9eb136492daf212a1d350060c1 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/b0d93c9eb136492daf212a1d350060c1 2023-04-19 07:56:22,857 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/c2b1ea84a35643eebcc23ad91e78cf8b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/C/c2b1ea84a35643eebcc23ad91e78cf8b 2023-04-19 07:56:22,859 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/recovered.edits/858.seqid to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15/recovered.edits/858.seqid 2023-04-19 07:56:22,859 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/199c6b7f1f3a9353227e2a51c7965b15 2023-04-19 07:56:22,859 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 07:56:22,861 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:22,862 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-19 07:56:22,863 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-19 07:56:22,864 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:22,864 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-19 07:56:22,864 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681890982864"}]},"ts":"9223372036854775807"} 2023-04-19 07:56:22,866 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-19 07:56:22,866 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 199c6b7f1f3a9353227e2a51c7965b15, NAME => 'TestAcidGuarantees,,1681890955073.199c6b7f1f3a9353227e2a51c7965b15.', STARTKEY => '', ENDKEY => ''}] 2023-04-19 07:56:22,866 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-19 07:56:22,866 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681890982866"}]},"ts":"9223372036854775807"} 2023-04-19 07:56:22,867 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-19 07:56:22,868 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:22,869 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 31 msec 2023-04-19 07:56:22,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-19 07:56:22,941 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-19 07:56:22,952 INFO [Listener at localhost/42517] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227 (was 226) - Thread LEAK? -, OpenFileDescriptor=553 (was 545) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=450 (was 462), ProcessCount=169 (was 169), AvailableMemoryMB=6650 (was 6531) - AvailableMemoryMB LEAK? - 2023-04-19 07:56:22,966 INFO [Listener at localhost/42517] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=227, OpenFileDescriptor=553, MaxFileDescriptor=60000, SystemLoadAverage=450, ProcessCount=169, AvailableMemoryMB=6649 2023-04-19 07:56:22,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 07:56:22,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 07:56:22,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:22,970 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-19 07:56:22,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-19 07:56:22,971 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-19 07:56:22,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-19 07:56:22,972 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:22,973 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2 empty. 2023-04-19 07:56:22,973 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:22,973 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 07:56:23,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-19 07:56:23,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-19 07:56:23,383 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 07:56:23,385 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 051c522ed965d63912b88bb381b9d1d2, NAME => 'TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp 2023-04-19 07:56:23,390 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:56:23,390 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 051c522ed965d63912b88bb381b9d1d2, disabling compactions & flushes 2023-04-19 07:56:23,390 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:23,390 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:23,391 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. after waiting 0 ms 2023-04-19 07:56:23,391 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:23,391 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:23,391 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:23,392 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-19 07:56:23,393 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890983393"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890983393"}]},"ts":"1681890983393"} 2023-04-19 07:56:23,394 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-19 07:56:23,395 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-19 07:56:23,395 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890983395"}]},"ts":"1681890983395"} 2023-04-19 07:56:23,396 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-19 07:56:23,399 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=051c522ed965d63912b88bb381b9d1d2, ASSIGN}] 2023-04-19 07:56:23,401 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=051c522ed965d63912b88bb381b9d1d2, ASSIGN 2023-04-19 07:56:23,401 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=051c522ed965d63912b88bb381b9d1d2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42359,1681890819191; forceNewPlan=false, retain=false 2023-04-19 07:56:23,553 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=051c522ed965d63912b88bb381b9d1d2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:23,553 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890983553"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890983553"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890983553"}]},"ts":"1681890983553"} 2023-04-19 07:56:23,554 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:56:23,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-19 07:56:23,710 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:23,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 051c522ed965d63912b88bb381b9d1d2, NAME => 'TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.', STARTKEY => '', ENDKEY => ''} 2023-04-19 07:56:23,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:23,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:56:23,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:23,711 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:23,712 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:23,714 DEBUG [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A 2023-04-19 07:56:23,714 DEBUG [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A 2023-04-19 07:56:23,714 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:56:23,715 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 051c522ed965d63912b88bb381b9d1d2 columnFamilyName A 2023-04-19 07:56:23,715 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.HStore(310): Store=051c522ed965d63912b88bb381b9d1d2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:56:23,715 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:23,716 DEBUG [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B 2023-04-19 07:56:23,716 DEBUG [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B 2023-04-19 07:56:23,717 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:56:23,717 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 051c522ed965d63912b88bb381b9d1d2 columnFamilyName B 2023-04-19 07:56:23,717 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.HStore(310): Store=051c522ed965d63912b88bb381b9d1d2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:56:23,718 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:23,719 DEBUG [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C 2023-04-19 07:56:23,719 DEBUG [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C 2023-04-19 07:56:23,719 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:56:23,719 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 051c522ed965d63912b88bb381b9d1d2 columnFamilyName C 2023-04-19 07:56:23,720 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.HStore(310): Store=051c522ed965d63912b88bb381b9d1d2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:56:23,720 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:23,721 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:23,721 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:23,723 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 07:56:23,724 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:23,726 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-19 07:56:23,727 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 051c522ed965d63912b88bb381b9d1d2; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75425850, jitterRate=0.12393274903297424}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 07:56:23,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:23,727 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., pid=56, masterSystemTime=1681890983705 2023-04-19 07:56:23,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:23,729 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:23,729 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=051c522ed965d63912b88bb381b9d1d2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:23,729 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890983729"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681890983729"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681890983729"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681890983729"}]},"ts":"1681890983729"} 2023-04-19 07:56:23,731 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-19 07:56:23,731 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 in 176 msec 2023-04-19 07:56:23,732 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-19 07:56:23,732 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=051c522ed965d63912b88bb381b9d1d2, ASSIGN in 332 msec 2023-04-19 07:56:23,733 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-19 07:56:23,733 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681890983733"}]},"ts":"1681890983733"} 2023-04-19 07:56:23,734 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-19 07:56:23,735 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-19 07:56:23,736 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 767 msec 2023-04-19 07:56:24,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-19 07:56:24,075 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-19 07:56:24,076 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x3a9f3972 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:56:24,081 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3011deb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:56:24,082 DEBUG [Listener at localhost/42517] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:56:24,083 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37880, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:56:24,084 DEBUG [Listener at localhost/42517] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-19 07:56:24,086 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42534, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-19 07:56:24,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-19 07:56:24,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-19 07:56:24,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:24,501 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-19 07:56:24,503 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-19 07:56:24,503 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-19 07:56:24,504 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-19 07:56:24,506 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=051c522ed965d63912b88bb381b9d1d2, REOPEN/MOVE}] 2023-04-19 07:56:24,507 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=051c522ed965d63912b88bb381b9d1d2, REOPEN/MOVE 2023-04-19 07:56:24,507 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=051c522ed965d63912b88bb381b9d1d2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:24,507 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890984507"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890984507"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890984507"}]},"ts":"1681890984507"} 2023-04-19 07:56:24,508 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:56:24,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:24,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 051c522ed965d63912b88bb381b9d1d2, disabling compactions & flushes 2023-04-19 07:56:24,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:24,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:24,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. after waiting 0 ms 2023-04-19 07:56:24,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:24,664 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-19 07:56:24,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:24,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:24,665 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 051c522ed965d63912b88bb381b9d1d2 to self. 2023-04-19 07:56:24,667 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:24,667 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=051c522ed965d63912b88bb381b9d1d2, regionState=CLOSED 2023-04-19 07:56:24,667 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890984667"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681890984667"}]},"ts":"1681890984667"} 2023-04-19 07:56:24,669 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-19 07:56:24,669 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 in 160 msec 2023-04-19 07:56:24,670 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=051c522ed965d63912b88bb381b9d1d2, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,42359,1681890819191; forceNewPlan=false, retain=true 2023-04-19 07:56:24,820 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=051c522ed965d63912b88bb381b9d1d2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:24,820 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890984820"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681890984820"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681890984820"}]},"ts":"1681890984820"} 2023-04-19 07:56:24,822 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:56:24,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:24,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 051c522ed965d63912b88bb381b9d1d2, NAME => 'TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.', STARTKEY => '', ENDKEY => ''} 2023-04-19 07:56:24,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:24,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-19 07:56:24,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:24,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:24,977 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:24,978 DEBUG [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A 2023-04-19 07:56:24,978 DEBUG [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A 2023-04-19 07:56:24,978 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:56:24,978 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 051c522ed965d63912b88bb381b9d1d2 columnFamilyName A 2023-04-19 07:56:24,979 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.HStore(310): Store=051c522ed965d63912b88bb381b9d1d2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:56:24,980 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:24,980 DEBUG [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B 2023-04-19 07:56:24,980 DEBUG [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B 2023-04-19 07:56:24,980 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:56:24,981 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 051c522ed965d63912b88bb381b9d1d2 columnFamilyName B 2023-04-19 07:56:24,981 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.HStore(310): Store=051c522ed965d63912b88bb381b9d1d2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:56:24,981 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:24,982 DEBUG [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C 2023-04-19 07:56:24,982 DEBUG [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C 2023-04-19 07:56:24,982 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-19 07:56:24,982 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 051c522ed965d63912b88bb381b9d1d2 columnFamilyName C 2023-04-19 07:56:24,982 INFO [StoreOpener-051c522ed965d63912b88bb381b9d1d2-1] regionserver.HStore(310): Store=051c522ed965d63912b88bb381b9d1d2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-19 07:56:24,982 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:24,983 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:24,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:24,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-19 07:56:24,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:24,987 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 051c522ed965d63912b88bb381b9d1d2; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59446013, jitterRate=-0.1141853779554367}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-19 07:56:24,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:24,988 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., pid=61, masterSystemTime=1681890984973 2023-04-19 07:56:24,989 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:24,989 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:24,989 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=051c522ed965d63912b88bb381b9d1d2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:24,990 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681890984989"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681890984989"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681890984989"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681890984989"}]},"ts":"1681890984989"} 2023-04-19 07:56:24,991 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-19 07:56:24,991 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 in 169 msec 2023-04-19 07:56:24,992 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-19 07:56:24,992 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=051c522ed965d63912b88bb381b9d1d2, REOPEN/MOVE in 485 msec 2023-04-19 07:56:24,994 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-19 07:56:24,994 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-04-19 07:56:24,995 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 906 msec 2023-04-19 07:56:24,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-19 07:56:24,997 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x2d9fccbf to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:56:25,002 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e8824d7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:56:25,003 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x3bafb41f to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:56:25,007 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68067de1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:56:25,007 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x03ced4ba to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:56:25,011 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45878790, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:56:25,011 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x63da10ca to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:56:25,015 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1644730f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:56:25,016 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x59185ba7 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:56:25,020 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6af74c0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:56:25,020 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x1d759aab to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:56:25,027 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4db22237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:56:25,028 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x39241fe3 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:56:25,033 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b57aae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:56:25,033 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbb5b46 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:56:25,041 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6799455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:56:25,041 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x23e65a76 to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:56:25,050 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66c25427, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:56:25,050 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(139): Connect 0x5e6a6f3c to 127.0.0.1:58087 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-19 07:56:25,053 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f4eab6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-19 07:56:25,064 DEBUG [hconnection-0xb7ad738-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:56:25,067 DEBUG [hconnection-0x23eef481-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:56:25,068 DEBUG [hconnection-0x6a32d518-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:56:25,068 DEBUG [hconnection-0xd306643-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:56:25,069 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37898, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:56:25,069 DEBUG [hconnection-0x6e412cfe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:56:25,069 DEBUG [hconnection-0x150faf2a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:56:25,070 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37930, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:56:25,073 DEBUG [hconnection-0x42b6a879-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:56:25,073 DEBUG [hconnection-0x73dd503f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:56:25,077 DEBUG [hconnection-0x5a0f2209-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:56:25,077 DEBUG [hconnection-0x2f45ce84-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-19 07:56:25,078 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37892, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:56:25,078 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37884, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:56:25,079 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37914, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:56:25,079 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37940, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:56:25,079 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37900, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:56:25,079 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37952, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:56:25,079 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37938, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:56:25,082 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37954, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-19 07:56:25,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:25,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:25,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:25,091 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:25,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:25,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:25,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:25,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,092 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:25,092 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:25,092 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,093 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:25,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:25,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:25,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:25,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:25,096 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:25,096 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:25,096 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:25,096 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:25,096 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:25,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:25,096 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:25,113 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-19 07:56:25,114 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:25,114 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:25,115 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:25,115 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:25,115 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:25,115 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:25,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:25,116 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:25,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890985122, max timestamp=1681890985132Num uniques 859; 2023-04-19 07:56:25,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890985122, max timestamp=1681890985132Num uniques 859; 2023-04-19 07:56:25,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:25,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:25,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890985122, max timestamp=1681890985133Num uniques 859; 2023-04-19 07:56:25,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:25,135 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304199f25178c314e4d9da5615e98c16b1fb7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199f25178c314e4d9da5615e98c16b1fb7 2023-04-19 07:56:25,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890985146, max timestamp=1681890985151Num uniques 1018; 2023-04-19 07:56:25,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681890985146, max timestamp=1681890985151Num uniques 1018; 2023-04-19 07:56:25,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:56:25,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7685, totalHeapSize=17712, min timestamp=1681890985146, max timestamp=1681890985151Num uniques 1018; 2023-04-19 07:56:25,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:56:25,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46615, new segment data size=7685, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:56:25,156 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=11, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/7c41184935594c28a5b080c2250534ff 2023-04-19 07:56:25,193 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/4e7009dc4f0840c6b2e2f9d3f3308ccc 2023-04-19 07:56:25,202 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/5380f65373eb4e07999569a56cc9057b 2023-04-19 07:56:25,205 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/7c41184935594c28a5b080c2250534ff as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/7c41184935594c28a5b080c2250534ff 2023-04-19 07:56:25,208 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/7c41184935594c28a5b080c2250534ff, entries=150, sequenceid=11, filesize=25.1 K 2023-04-19 07:56:25,209 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/4e7009dc4f0840c6b2e2f9d3f3308ccc as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4e7009dc4f0840c6b2e2f9d3f3308ccc 2023-04-19 07:56:25,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681890985157, max timestamp=1681890985187Num uniques 1027; 2023-04-19 07:56:25,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-19 07:56:25,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681890985157, max timestamp=1681890985187Num uniques 1027; 2023-04-19 07:56:25,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-19 07:56:25,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681890985163, max timestamp=1681890985187Num uniques 1027; 2023-04-19 07:56:25,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-19 07:56:25,214 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4e7009dc4f0840c6b2e2f9d3f3308ccc, entries=150, sequenceid=11, filesize=11.6 K 2023-04-19 07:56:25,214 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/5380f65373eb4e07999569a56cc9057b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5380f65373eb4e07999569a56cc9057b 2023-04-19 07:56:25,217 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5380f65373eb4e07999569a56cc9057b, entries=150, sequenceid=11, filesize=11.6 K 2023-04-19 07:56:25,218 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=74.83 KB/76628 for 051c522ed965d63912b88bb381b9d1d2 in 121ms, sequenceid=11, compaction requested=false 2023-04-19 07:56:25,218 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-19 07:56:25,219 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:25,219 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:25,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:25,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:25,219 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:25,220 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:25,221 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,221 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,222 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:25,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:25,224 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=88.25 KB heapSize=241.49 KB 2023-04-19 07:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@656377f3[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:25,224 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:25,224 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:25,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:25,226 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,227 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:25,227 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:25,227 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,227 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:25,227 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:25,228 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:25,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:25,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:25,232 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:25,232 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:25,232 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:25,232 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:25,232 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:25,233 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,233 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:25,233 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,234 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,236 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,239 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:25,239 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:25,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:25,239 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:25,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:25,239 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,239 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,239 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,239 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,239 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,239 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,239 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:25,240 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:25,239 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:25,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:25,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891045238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,247 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419eb6bd23a70f8444e9f0d76e666bb37eb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419eb6bd23a70f8444e9f0d76e666bb37eb 2023-04-19 07:56:25,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891045242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891045243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891045244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891045244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,340 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:25,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:25,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:25,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:25,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:25,343 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:25,343 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:25,343 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:25,343 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:25,343 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,343 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,346 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,346 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,346 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:25,346 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:25,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:25,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:25,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:25,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:25,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:25,348 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:25,348 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:25,348 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:25,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,348 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:25,348 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:25,348 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:25,348 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:25,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891045344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,348 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:25,348 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:25,349 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:25,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891045349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891045349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891045350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,353 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891045349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,353 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,353 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,353 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:25,355 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:25,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c32a21f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:25,355 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,356 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,356 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:25,356 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:25,356 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:25,356 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,356 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:25,359 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,359 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,359 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:25,359 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:25,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:25,361 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:25,361 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:25,361 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:25,361 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:25,361 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:25,362 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,363 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,363 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:25,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,363 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,364 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,367 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,367 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:25,367 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:25,367 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,367 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:25,367 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:25,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:25,367 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:25,367 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:25,367 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:25,367 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:25,367 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,367 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:25,467 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:25,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:25,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:25,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:25,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:25,471 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:25,471 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:25,471 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:25,471 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:25,472 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,472 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,473 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,473 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:25,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:25,473 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,473 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:25,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:25,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:25,474 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:25,475 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:25,475 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:25,475 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:25,475 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:25,477 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,477 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,478 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,478 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:25,479 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:25,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2773877f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:25,479 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,480 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:25,481 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:25,481 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,481 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:25,482 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:25,482 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,482 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,482 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:25,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:25,484 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:25,484 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:25,484 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:25,484 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:25,484 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:25,486 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,486 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,486 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:25,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:25,486 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,488 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,490 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,491 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,491 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,491 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:25,491 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:25,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:25,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:25,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:25,491 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:25,491 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,491 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,491 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,491 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:25,491 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:25,491 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:25,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:25,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891045550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891045553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891045554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891045554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891045555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,591 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:25,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:25,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:25,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:25,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:25,595 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:25,595 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:25,595 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:25,595 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:25,595 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,595 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,596 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:25,596 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:25,597 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,597 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:25,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:25,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:25,598 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:25,598 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:25,598 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:25,598 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:25,598 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:25,598 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:25,598 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:25,598 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:25,598 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:25,598 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:25,598 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,600 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:25,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:25,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@149a24e7[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:25,602 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:25,602 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,603 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,603 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:25,603 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:25,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:25,603 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:25,606 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:25,606 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:25,606 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:25,606 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:25,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:25,609 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:25,609 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:25,609 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:25,609 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:25,609 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:25,610 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,610 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,610 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,611 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,612 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,614 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,614 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:25,614 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:25,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:25,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:25,614 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:25,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:25,614 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:25,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:25,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,615 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:25,615 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:25,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,615 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,654 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=77, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/9e1b51b6031f413aa0a68f92816b7618 2023-04-19 07:56:25,715 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:25,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:25,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:25,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:25,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:25,719 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:25,719 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:25,719 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:25,719 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:25,720 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,720 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,721 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:25,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:25,721 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,722 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,722 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:25,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:25,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:25,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:25,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:25,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:25,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:25,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:25,724 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:25,724 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:25,724 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:25,724 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:25,725 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:25,725 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:25,725 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,725 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:25,725 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:25,726 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,727 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,727 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:25,728 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:25,728 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ab80568[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:25,729 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,730 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:25,730 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:25,730 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,730 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:25,730 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:25,735 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,735 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:25,735 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,735 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:25,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:25,737 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:25,737 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:25,737 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:25,737 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:25,737 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:25,739 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,739 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,739 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:25,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,748 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,750 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,752 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,752 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,752 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,752 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,752 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,752 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,752 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:25,752 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:25,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:25,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:25,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:25,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:25,752 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:25,752 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,752 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:25,752 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:25,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:25,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:25,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:25,852 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:25,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:25,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:25,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:25,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:25,856 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:25,856 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:25,856 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:25,856 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:25,856 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,856 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891045854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891045855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,858 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:25,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:25,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:25,858 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,858 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,859 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:25,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891045856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891045857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:25,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:25,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:25,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891045858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:25,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:25,860 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:25,860 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:25,860 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:25,860 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:25,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:25,862 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,862 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,863 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,863 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:25,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:25,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6393707a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:25,864 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:25,864 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,865 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:25,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:25,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:25,866 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,866 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:25,867 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,867 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:25,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,867 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:25,867 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:25,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:25,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:25,870 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:25,870 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:25,870 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:25,870 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:25,870 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:25,871 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,871 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,871 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:25,872 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,874 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,877 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,877 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:25,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:25,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:25,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:25,877 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:25,877 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,877 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,878 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,878 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,878 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,878 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,877 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:25,878 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:25,878 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:25,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:25,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:25,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:25,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:25,978 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:25,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:25,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:25,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:25,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:25,981 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:25,981 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:25,981 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:25,981 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:25,982 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,982 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,983 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,984 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:25,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:25,985 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,985 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:25,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:25,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:25,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:25,986 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:25,986 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:25,986 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:25,986 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,986 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:25,987 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:25,988 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,988 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,989 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,989 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:25,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58e04b08[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:25,990 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:25,990 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,991 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:25,991 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,991 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:25,991 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:25,993 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,993 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:25,994 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:25,994 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:25,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:25,996 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:25,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:25,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:25,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:25,996 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:25,998 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,998 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,998 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:25,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:25,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,000 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:26,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,003 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:26,006 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:26,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:26,006 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:26,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:26,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:26,006 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:26,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:26,007 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:26,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:26,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:26,064 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/16efe36d98e147f1a596e6bec27ada5a 2023-04-19 07:56:26,107 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:26,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:26,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:26,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:26,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:26,110 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:26,110 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:26,110 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:26,110 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:26,110 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,110 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,112 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:26,112 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:26,112 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,112 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:26,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:26,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:26,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:26,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:26,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:26,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:26,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:26,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:26,114 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:26,114 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:26,114 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:26,114 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:26,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:26,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:26,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:26,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:26,117 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,117 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:26,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:26,117 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,117 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:26,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:26,119 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:26,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d18e1ef[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:26,119 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,120 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,120 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:26,120 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:26,120 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:26,121 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,121 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:26,122 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,122 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:26,122 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:26,122 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:26,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:26,125 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:26,125 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:26,125 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:26,125 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:26,125 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:26,129 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,129 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:26,129 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:26,130 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,131 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:26,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:26,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:26,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:26,134 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:26,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:26,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:26,134 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,134 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:26,135 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:26,135 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,135 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,135 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,135 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,135 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:26,235 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:26,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:26,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:26,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:26,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:26,238 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:26,238 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:26,238 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:26,238 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:26,239 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,239 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,240 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:26,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:26,240 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,240 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,240 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:26,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:26,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:26,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:26,241 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:26,241 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:26,241 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:26,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:26,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:26,242 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:26,242 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:26,242 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:26,242 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:26,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:26,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:26,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:26,242 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:26,243 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,243 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:26,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:26,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:26,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:26,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:26,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37213461[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:26,245 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:26,245 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,247 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:26,248 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:26,248 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,248 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:26,249 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:26,249 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:26,249 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:26,249 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:26,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:26,251 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:26,251 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:26,252 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:26,252 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:26,252 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:26,253 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,253 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:26,253 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:26,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:26,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:26,254 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,255 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,257 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,257 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,257 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:26,257 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:26,257 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:26,257 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,257 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:26,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:26,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:26,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:26,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,258 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,258 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:26,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:26,258 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:26,358 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:26,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:26,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:26,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:26,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:26,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:26,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891046360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,362 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:26,362 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:26,362 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:26,362 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:26,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:26,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891046361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,362 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,362 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:26,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:26,365 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:26,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891046362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,365 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891046362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:26,365 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891046364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,365 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:26,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:26,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:26,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:26,366 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:26,366 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:26,366 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:26,366 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:26,366 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:26,367 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:26,367 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:26,367 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:26,367 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:26,367 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:26,367 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:26,367 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,367 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:26,367 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:26,368 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,368 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:26,369 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,369 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:26,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:26,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fc9925f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:26,370 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:26,370 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,371 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,371 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:26,371 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:26,371 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,372 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:26,371 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:26,373 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:26,373 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,373 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:26,373 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:26,375 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:26,375 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:26,375 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:26,376 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:26,376 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:26,378 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,378 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,378 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:26,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:26,379 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:26,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,381 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:26,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,383 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,383 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,383 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,383 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:26,383 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:26,383 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,383 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,383 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:26,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:26,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:26,383 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,383 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:26,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:26,383 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:26,383 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:26,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:26,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=32.34 KB at sequenceid=77 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/3ff0968a9f394aabab0e32ca0d727815 2023-04-19 07:56:26,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/9e1b51b6031f413aa0a68f92816b7618 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/9e1b51b6031f413aa0a68f92816b7618 2023-04-19 07:56:26,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/9e1b51b6031f413aa0a68f92816b7618, entries=150, sequenceid=77, filesize=25.1 K 2023-04-19 07:56:26,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/16efe36d98e147f1a596e6bec27ada5a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/16efe36d98e147f1a596e6bec27ada5a 2023-04-19 07:56:26,483 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:26,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:26,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:26,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/16efe36d98e147f1a596e6bec27ada5a, entries=150, sequenceid=77, filesize=11.6 K 2023-04-19 07:56:26,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:26,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:26,489 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:26,489 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:26,489 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:26,489 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:26,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/3ff0968a9f394aabab0e32ca0d727815 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/3ff0968a9f394aabab0e32ca0d727815 2023-04-19 07:56:26,490 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,490 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,491 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:26,491 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,492 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,492 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:26,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:26,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:26,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:26,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:26,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:26,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:26,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:26,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:26,493 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:26,493 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:26,493 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:26,493 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:26,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:26,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:26,493 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,494 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:26,494 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:26,495 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:26,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:26,497 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:26,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@152b28a5[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:26,497 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,497 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/3ff0968a9f394aabab0e32ca0d727815, entries=150, sequenceid=77, filesize=11.6 K 2023-04-19 07:56:26,498 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~108.38 KB/110978, heapSize ~301.21 KB/308440, currentSize=87.22 KB/89310 for 051c522ed965d63912b88bb381b9d1d2 in 1274ms, sequenceid=77, compaction requested=false 2023-04-19 07:56:26,499 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:26,499 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,500 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:26,500 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:26,500 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:26,500 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,500 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:26,501 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:26,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:26,504 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:26,504 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:26,504 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:26,504 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:26,504 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:26,505 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,505 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:26,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,505 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:26,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,507 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:26,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,513 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:26,513 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:26,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:26,514 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:26,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:26,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:26,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:26,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:26,514 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,514 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,514 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,514 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,514 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,514 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:26,514 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:26,514 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:26,514 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,614 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:26,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:26,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:26,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:26,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:26,618 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:26,618 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:26,618 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:26,618 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:26,618 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,618 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,619 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,619 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:26,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:26,620 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:26,620 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:26,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:26,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:26,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:26,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:26,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:26,621 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:26,621 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:26,621 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-19 07:56:26,622 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:26,622 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:26,622 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:26,622 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:26,622 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:26,622 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:26,637 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041991782e7f6581408cbafa2590cd29a15c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041991782e7f6581408cbafa2590cd29a15c 2023-04-19 07:56:27,042 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=93, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/8fbfc346eadb4648948e701719685988 2023-04-19 07:56:27,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:27,364 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:27,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891047375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891047376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891047377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891047378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891047378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,452 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/74dabad365af4b0dbd8118e3155541a8 2023-04-19 07:56:27,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891047480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891047481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891047481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891047482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891047482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891047684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891047684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891047684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891047686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:27,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891047686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,854 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-19 07:56:27,861 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/f12abd94895e4d31b0014b7e4131653a 2023-04-19 07:56:27,867 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/8fbfc346eadb4648948e701719685988 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/8fbfc346eadb4648948e701719685988 2023-04-19 07:56:27,872 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/8fbfc346eadb4648948e701719685988, entries=150, sequenceid=93, filesize=25.1 K 2023-04-19 07:56:27,872 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/74dabad365af4b0dbd8118e3155541a8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/74dabad365af4b0dbd8118e3155541a8 2023-04-19 07:56:27,876 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/74dabad365af4b0dbd8118e3155541a8, entries=150, sequenceid=93, filesize=11.6 K 2023-04-19 07:56:27,877 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/f12abd94895e4d31b0014b7e4131653a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/f12abd94895e4d31b0014b7e4131653a 2023-04-19 07:56:27,881 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/f12abd94895e4d31b0014b7e4131653a, entries=150, sequenceid=93, filesize=11.6 K 2023-04-19 07:56:27,882 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 051c522ed965d63912b88bb381b9d1d2 in 1261ms, sequenceid=93, compaction requested=true 2023-04-19 07:56:27,882 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:27,882 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:27,882 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:27,882 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:27,882 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,882 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:27,882 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:27,884 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:27,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:27,884 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fe6a476[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:27,886 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:27,886 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:27,889 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:27,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:27,889 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:27,889 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:27,889 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:27,889 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:27,889 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:27,889 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,889 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:27,890 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:27,890 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,890 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:27,891 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:27,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:27,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:27,895 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:27,895 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:27,895 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:27,895 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:27,895 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:27,897 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:27,897 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:27,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:27,897 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:27,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:27,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:27,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:27,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:27,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:27,898 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:27,900 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:27,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,902 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:27,902 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:27,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:27,903 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,903 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:27,903 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:27,903 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:27,903 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:27,903 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:27,903 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:27,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:27,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:27,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:27,903 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:27,903 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:27,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:27,903 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:27,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:27,989 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-19 07:56:27,989 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:27,990 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:27,990 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:27,990 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:27,990 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:27,990 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:28,003 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:28,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:28,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891048001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:28,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891048002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891048003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:28,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:28,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891048003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,009 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:28,009 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:28,009 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:28,009 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:28,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891048005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,010 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,010 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,011 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:28,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:28,012 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,012 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,012 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:28,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:28,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:28,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:28,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:28,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:28,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:28,029 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:28,029 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:28,029 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,029 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:28,030 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:28,030 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,030 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:28,030 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:28,030 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,030 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:28,030 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:28,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,033 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,033 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,033 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71da03d7[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:28,036 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:28,036 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,037 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,037 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:28,037 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:28,037 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:28,037 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,038 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:28,039 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,039 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:28,039 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:28,039 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:28,042 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:28,042 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:28,042 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:28,042 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:28,042 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:28,045 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,045 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:28,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,045 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,046 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,050 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:28,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:28,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:28,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:28,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:28,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:28,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:28,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:28,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,050 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:28,050 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:28,050 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,051 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,051 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:28,051 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:28,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891048107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891048108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891048110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891048110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891048111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,151 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:28,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:28,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:28,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:28,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:28,154 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:28,154 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:28,154 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:28,154 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:28,155 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,155 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,156 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:28,156 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:28,157 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,157 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:28,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:28,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:28,158 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:28,158 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,158 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:28,158 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:28,159 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,159 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:28,159 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:28,159 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,159 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:28,159 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:28,160 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,161 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,161 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,161 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:28,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:28,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72823cd9[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:28,163 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:28,163 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,165 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,165 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:28,165 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:28,165 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:28,166 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,166 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:28,167 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,167 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:28,167 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,167 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:28,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:28,171 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:28,171 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:28,171 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:28,171 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:28,171 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:28,173 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,173 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,173 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:28,174 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,175 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,178 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,178 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,178 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,178 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:28,178 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:28,178 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:28,178 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:28,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:28,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:28,178 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,178 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:28,178 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:28,178 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,178 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:28,278 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:28,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:28,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:28,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:28,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:28,281 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:28,282 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:28,282 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:28,282 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:28,283 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,283 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,288 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,288 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:28,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:28,288 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,288 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:28,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:28,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:28,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:28,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:28,290 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:28,290 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:28,290 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:28,290 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:28,290 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:28,290 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,290 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:28,290 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:28,290 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,291 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:28,291 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:28,291 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,291 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:28,291 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:28,292 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,292 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,293 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,293 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:28,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:28,294 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:28,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ea05c2c[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:28,294 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,295 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:28,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:28,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:28,296 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,296 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:28,298 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,298 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:28,298 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,298 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:28,301 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:28,301 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:28,301 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:28,301 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:28,301 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:28,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:28,303 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,305 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:28,307 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:28,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:28,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:28,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:28,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:28,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:28,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,308 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,308 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,308 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,308 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,308 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:28,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:28,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:28,309 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:28,309 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:28,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891048311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891048312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891048315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891048315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891048315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,409 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:28,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:28,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:28,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:28,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:28,414 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:28,414 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:28,414 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:28,414 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:28,414 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,414 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,415 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:28,415 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:28,416 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,416 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:28,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:28,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:28,417 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:28,417 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:28,417 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:28,417 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:28,417 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:28,417 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,417 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:28,417 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:28,417 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:28,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:28,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:28,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:28,420 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,420 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,420 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,420 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:28,422 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:28,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449f8a9a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:28,422 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,423 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:28,423 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,423 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:28,425 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:28,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,425 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:28,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:28,427 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:28,427 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:28,427 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:28,427 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:28,427 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:28,429 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,429 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,429 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:28,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:28,429 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,431 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,436 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419e6828bc897ed4f96b4cb00fe2bf55572 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e6828bc897ed4f96b4cb00fe2bf55572 2023-04-19 07:56:28,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:28,436 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,436 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,436 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:28,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:28,436 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:28,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:28,437 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:28,437 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:28,436 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:28,437 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:28,436 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,437 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,437 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,437 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:28,441 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=111, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/5f4b234f704f4a6ca40eb79d503d250f 2023-04-19 07:56:28,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/9af8c2e161a141af964e163a4ab9ea30 2023-04-19 07:56:28,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/beb609c6a6234e73bf274807ebaeaf87 2023-04-19 07:56:28,464 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/5f4b234f704f4a6ca40eb79d503d250f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/5f4b234f704f4a6ca40eb79d503d250f 2023-04-19 07:56:28,467 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/5f4b234f704f4a6ca40eb79d503d250f, entries=200, sequenceid=111, filesize=31.9 K 2023-04-19 07:56:28,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/9af8c2e161a141af964e163a4ab9ea30 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9af8c2e161a141af964e163a4ab9ea30 2023-04-19 07:56:28,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9af8c2e161a141af964e163a4ab9ea30, entries=150, sequenceid=111, filesize=11.6 K 2023-04-19 07:56:28,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/beb609c6a6234e73bf274807ebaeaf87 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/beb609c6a6234e73bf274807ebaeaf87 2023-04-19 07:56:28,476 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/beb609c6a6234e73bf274807ebaeaf87, entries=150, sequenceid=111, filesize=11.6 K 2023-04-19 07:56:28,476 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 051c522ed965d63912b88bb381b9d1d2 in 487ms, sequenceid=111, compaction requested=true 2023-04-19 07:56:28,476 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:28,476 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:28,477 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:28,477 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:56:28,477 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:28,477 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:56:28,478 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 109924 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:56:28,478 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/A is initiating minor compaction (all files) 2023-04-19 07:56:28,478 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/A in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,478 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/7c41184935594c28a5b080c2250534ff, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/9e1b51b6031f413aa0a68f92816b7618, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/8fbfc346eadb4648948e701719685988, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/5f4b234f704f4a6ca40eb79d503d250f] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=107.3 K 2023-04-19 07:56:28,478 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:56:28,478 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/B is initiating minor compaction (all files) 2023-04-19 07:56:28,478 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/B in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,478 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4e7009dc4f0840c6b2e2f9d3f3308ccc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/16efe36d98e147f1a596e6bec27ada5a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/74dabad365af4b0dbd8118e3155541a8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9af8c2e161a141af964e163a4ab9ea30] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=46.4 K 2023-04-19 07:56:28,479 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 7c41184935594c28a5b080c2250534ff, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681890985082 2023-04-19 07:56:28,479 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 4e7009dc4f0840c6b2e2f9d3f3308ccc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681890985082 2023-04-19 07:56:28,479 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 9e1b51b6031f413aa0a68f92816b7618, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1681890985216 2023-04-19 07:56:28,479 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 16efe36d98e147f1a596e6bec27ada5a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1681890985216 2023-04-19 07:56:28,480 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 8fbfc346eadb4648948e701719685988, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681890985228 2023-04-19 07:56:28,480 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 74dabad365af4b0dbd8118e3155541a8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681890985228 2023-04-19 07:56:28,480 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 5f4b234f704f4a6ca40eb79d503d250f, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681890987376 2023-04-19 07:56:28,480 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 9af8c2e161a141af964e163a4ab9ea30, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681890987376 2023-04-19 07:56:28,487 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#B#compaction#510 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:28,491 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#A#compaction#511 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:28,537 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:28,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:28,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:28,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:28,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:28,541 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:28,541 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:28,541 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:28,541 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:28,542 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,542 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,543 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:28,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:28,543 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,544 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,544 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:28,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:28,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:28,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:28,545 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:28,545 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:28,545 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:28,545 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:28,545 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:28,545 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,545 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:28,545 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-19 07:56:28,545 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:28,545 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:28,545 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:28,545 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:28,546 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:28,546 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:28,565 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041926614aefca8b4a1db2958228699f5195 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041926614aefca8b4a1db2958228699f5195 2023-04-19 07:56:28,573 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=125, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/61dcfacfd4a841499aad606971c9d258 2023-04-19 07:56:28,583 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/a8256069426649ad9bd5be0cb9a9256c 2023-04-19 07:56:28,601 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/1d0d01f3e10f44a2bce2def1d0c05a88 2023-04-19 07:56:28,606 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/61dcfacfd4a841499aad606971c9d258 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/61dcfacfd4a841499aad606971c9d258 2023-04-19 07:56:28,611 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/61dcfacfd4a841499aad606971c9d258, entries=150, sequenceid=125, filesize=25.1 K 2023-04-19 07:56:28,613 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/a8256069426649ad9bd5be0cb9a9256c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/a8256069426649ad9bd5be0cb9a9256c 2023-04-19 07:56:28,616 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/a8256069426649ad9bd5be0cb9a9256c, entries=150, sequenceid=125, filesize=11.6 K 2023-04-19 07:56:28,617 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/1d0d01f3e10f44a2bce2def1d0c05a88 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/1d0d01f3e10f44a2bce2def1d0c05a88 2023-04-19 07:56:28,629 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/1d0d01f3e10f44a2bce2def1d0c05a88, entries=150, sequenceid=125, filesize=11.6 K 2023-04-19 07:56:28,630 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 051c522ed965d63912b88bb381b9d1d2 in 85ms, sequenceid=125, compaction requested=true 2023-04-19 07:56:28,630 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:28,630 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:28,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:28,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:28,630 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:28,632 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,632 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,632 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,632 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:28,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65faed65[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:28,634 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:28,634 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,635 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,635 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:28,635 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:28,635 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:28,636 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,636 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:28,637 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:28,637 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:28,637 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,637 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:28,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:28,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:28,640 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:28,640 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:28,640 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:28,640 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:28,640 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:28,642 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,642 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,642 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:28,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,642 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,644 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,646 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:28,646 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,646 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:28,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:28,646 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:28,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:28,647 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:28,647 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:28,646 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,647 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,647 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,646 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:28,647 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,647 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:28,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:28,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:28,664 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:56:28,664 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:28,664 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:28,664 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:28,664 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:28,664 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:28,664 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:28,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890988674, max timestamp=1681890988690Num uniques 859; 2023-04-19 07:56:28,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:28,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890988674, max timestamp=1681890988690Num uniques 859; 2023-04-19 07:56:28,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:28,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890988674, max timestamp=1681890988690Num uniques 859; 2023-04-19 07:56:28,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:28,703 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041975fad4e26c404e4db3074eb94fb86021 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041975fad4e26c404e4db3074eb94fb86021 2023-04-19 07:56:28,718 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=135, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/10b12855608d434d8195ed48fe7ffc96 2023-04-19 07:56:28,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891048712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681890988705, max timestamp=1681890988718Num uniques 1018; 2023-04-19 07:56:28,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681890988705, max timestamp=1681890988718Num uniques 1018; 2023-04-19 07:56:28,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-19 07:56:28,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-19 07:56:28,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681890988705, max timestamp=1681890988718Num uniques 1018; 2023-04-19 07:56:28,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-19 07:56:28,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891048740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891048741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891048742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891048743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,747 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:28,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:28,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:28,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:28,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:28,750 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:28,750 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:28,750 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:28,750 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:28,750 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,750 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,751 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:28,751 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:28,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:28,752 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,752 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:28,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:28,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:28,756 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:28,756 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:28,756 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:28,758 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:28,758 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:28,758 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,758 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:28,758 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:28,758 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,758 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:28,758 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:28,758 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,758 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:28,758 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:28,764 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,764 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,765 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,765 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:28,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fd71f7[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:28,766 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:28,766 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,767 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,768 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:28,768 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:28,768 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,768 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:28,768 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:28,769 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,769 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:28,769 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:28,769 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:28,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:28,774 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:28,774 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:28,774 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:28,774 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:28,774 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:28,777 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,777 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,777 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:28,777 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,779 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,781 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,781 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,781 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,781 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,781 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,781 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:28,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:28,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:28,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:28,781 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:28,781 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:28,781 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:28,781 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:28,781 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:28,781 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:28,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:28,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:28,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891048820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891048846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891048846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891048847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:28,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891048847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,881 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:28,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:28,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:28,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:28,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:28,885 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:28,885 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:28,885 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:28,885 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:28,885 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,885 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,886 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:28,886 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:28,887 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,887 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:28,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:28,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:28,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:28,888 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:28,888 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:28,888 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:28,888 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:28,888 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:28,888 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,888 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:28,888 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:28,889 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:28,890 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,890 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,890 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,891 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:28,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:28,892 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:28,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a800c38[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:28,892 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,894 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,894 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:28,894 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:28,894 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:28,894 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,894 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:28,895 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,895 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:28,896 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:28,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:28,899 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:28,899 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:28,899 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:28,899 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:28,899 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:28,900 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,900 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,900 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:28,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:28,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:28,901 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:28,902 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/4f59fd14cdd542fc91392e8ffd17d482 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4f59fd14cdd542fc91392e8ffd17d482 2023-04-19 07:56:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:28,903 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:28,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:28,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,910 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:28,910 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:28,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:28,910 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/B of 051c522ed965d63912b88bb381b9d1d2 into 4f59fd14cdd542fc91392e8ffd17d482(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:28,910 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,911 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:28,911 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,911 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:28,911 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:28,911 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,911 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:28,911 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:28,911 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:28,911 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/B, priority=12, startTime=1681890988477; duration=0sec 2023-04-19 07:56:28,911 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:28,911 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:56:28,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:28,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:28,912 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:28,912 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:28,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:28,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:28,913 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:56:28,913 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/C is initiating minor compaction (all files) 2023-04-19 07:56:28,913 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/C in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:28,913 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5380f65373eb4e07999569a56cc9057b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/3ff0968a9f394aabab0e32ca0d727815, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/f12abd94895e4d31b0014b7e4131653a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/beb609c6a6234e73bf274807ebaeaf87, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/1d0d01f3e10f44a2bce2def1d0c05a88] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=58.0 K 2023-04-19 07:56:28,914 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 5380f65373eb4e07999569a56cc9057b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681890985082 2023-04-19 07:56:28,915 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 3ff0968a9f394aabab0e32ca0d727815, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=77, earliestPutTs=1681890985216 2023-04-19 07:56:28,915 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting f12abd94895e4d31b0014b7e4131653a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681890985228 2023-04-19 07:56:28,915 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting beb609c6a6234e73bf274807ebaeaf87, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681890987376 2023-04-19 07:56:28,916 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 1d0d01f3e10f44a2bce2def1d0c05a88, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681890987999 2023-04-19 07:56:28,919 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/68af14c5a04748deb6f35c23028124d5 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/68af14c5a04748deb6f35c23028124d5 2023-04-19 07:56:28,925 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/A of 051c522ed965d63912b88bb381b9d1d2 into 68af14c5a04748deb6f35c23028124d5(size=25.3 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:28,925 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:28,925 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/A, priority=12, startTime=1681890988476; duration=0sec 2023-04-19 07:56:28,925 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:28,927 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#C#compaction#517 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:28,935 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/261e17d077c546358c96151c42782348 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/261e17d077c546358c96151c42782348 2023-04-19 07:56:28,938 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/C of 051c522ed965d63912b88bb381b9d1d2 into 261e17d077c546358c96151c42782348(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:28,938 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:28,938 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/C, priority=11, startTime=1681890988477; duration=0sec 2023-04-19 07:56:28,939 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:29,012 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:29,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:29,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:29,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:29,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:29,015 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:29,015 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:29,015 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:29,015 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:29,016 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,016 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,018 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:29,018 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:29,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:29,018 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,019 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:29,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:29,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:29,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:29,020 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,020 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:29,020 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:29,020 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:29,020 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:29,022 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,022 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,022 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,022 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:29,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2739aa8[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:29,024 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:29,024 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,025 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,025 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:29,025 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:29,025 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:29,025 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,025 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:29,026 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:29,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891049024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:29,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:29,029 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:29,030 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:29,030 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:29,030 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:29,030 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:29,032 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,033 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,033 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:29,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,033 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,035 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,037 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:29,037 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:29,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:29,037 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:29,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:29,037 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:29,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:29,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,038 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:29,038 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:29,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:29,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891049050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891049051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891049051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891049051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/1681dc94d782439eaea2e08b2b97ec41 2023-04-19 07:56:29,138 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:29,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:29,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:29,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:29,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:29,142 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:29,142 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:29,142 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:29,142 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:29,142 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,142 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,143 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/ca853285a6204f7997e3e2cbe8047737 2023-04-19 07:56:29,147 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/10b12855608d434d8195ed48fe7ffc96 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/10b12855608d434d8195ed48fe7ffc96 2023-04-19 07:56:29,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/10b12855608d434d8195ed48fe7ffc96, entries=350, sequenceid=135, filesize=52.6 K 2023-04-19 07:56:29,151 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/1681dc94d782439eaea2e08b2b97ec41 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1681dc94d782439eaea2e08b2b97ec41 2023-04-19 07:56:29,153 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:29,153 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,154 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,154 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:29,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1681dc94d782439eaea2e08b2b97ec41, entries=150, sequenceid=135, filesize=11.8 K 2023-04-19 07:56:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:29,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:29,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/ca853285a6204f7997e3e2cbe8047737 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/ca853285a6204f7997e3e2cbe8047737 2023-04-19 07:56:29,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:29,155 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:29,155 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:29,155 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:29,155 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:29,155 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:29,155 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,155 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:29,156 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:29,156 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,156 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:29,156 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:29,156 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,156 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:29,156 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:29,157 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,157 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,158 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/ca853285a6204f7997e3e2cbe8047737, entries=150, sequenceid=135, filesize=11.8 K 2023-04-19 07:56:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,158 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,159 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 051c522ed965d63912b88bb381b9d1d2 in 495ms, sequenceid=135, compaction requested=true 2023-04-19 07:56:29,159 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,159 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:29,159 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:29,159 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:29,159 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:29,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:29,159 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:29,160 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:29,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3050ce46[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:29,160 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,160 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105490 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:29,160 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/A is initiating minor compaction (all files) 2023-04-19 07:56:29,160 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:29,160 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/A in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,161 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/B is initiating minor compaction (all files) 2023-04-19 07:56:29,161 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/68af14c5a04748deb6f35c23028124d5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/61dcfacfd4a841499aad606971c9d258, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/10b12855608d434d8195ed48fe7ffc96] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=103.0 K 2023-04-19 07:56:29,161 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/B in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,161 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4f59fd14cdd542fc91392e8ffd17d482, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/a8256069426649ad9bd5be0cb9a9256c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1681dc94d782439eaea2e08b2b97ec41] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=35.1 K 2023-04-19 07:56:29,161 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 68af14c5a04748deb6f35c23028124d5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681890987376 2023-04-19 07:56:29,161 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 4f59fd14cdd542fc91392e8ffd17d482, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1681890987376 2023-04-19 07:56:29,162 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,162 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:29,162 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 61dcfacfd4a841499aad606971c9d258, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681890987999 2023-04-19 07:56:29,162 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:29,162 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting a8256069426649ad9bd5be0cb9a9256c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681890987999 2023-04-19 07:56:29,162 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,162 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:29,162 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:29,162 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 10b12855608d434d8195ed48fe7ffc96, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681890988646 2023-04-19 07:56:29,163 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 1681dc94d782439eaea2e08b2b97ec41, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681890988649 2023-04-19 07:56:29,166 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:29,166 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:29,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:29,169 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:29,169 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:29,169 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:29,169 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:29,169 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#B#compaction#519 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:29,169 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:29,175 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,175 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,175 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:29,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:29,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:29,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,176 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,177 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,180 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,180 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,180 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,180 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:29,180 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:29,180 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:29,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:29,181 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:29,181 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:29,180 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,181 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,181 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:29,181 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:29,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:29,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:29,204 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#A#compaction#520 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:29,207 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/10cfcfd8fe4a4fd495ba7e94bdfa80df as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/10cfcfd8fe4a4fd495ba7e94bdfa80df 2023-04-19 07:56:29,211 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/B of 051c522ed965d63912b88bb381b9d1d2 into 10cfcfd8fe4a4fd495ba7e94bdfa80df(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:29,211 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:29,211 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/B, priority=13, startTime=1681890989159; duration=0sec 2023-04-19 07:56:29,211 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:29,211 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-19 07:56:29,212 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-19 07:56:29,212 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-19 07:56:29,212 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. because compaction request was cancelled 2023-04-19 07:56:29,229 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/f60f9baca9dc444caf34f9dad2c83e8a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/f60f9baca9dc444caf34f9dad2c83e8a 2023-04-19 07:56:29,233 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/A of 051c522ed965d63912b88bb381b9d1d2 into f60f9baca9dc444caf34f9dad2c83e8a(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:29,233 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:29,233 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/A, priority=13, startTime=1681890989159; duration=0sec 2023-04-19 07:56:29,233 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:29,281 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:29,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:29,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:29,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:29,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:29,284 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:29,284 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:29,284 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:29,284 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:29,285 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,285 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,287 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:29,287 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:29,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:29,287 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,287 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:29,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:29,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:29,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:29,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:29,288 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,289 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:29,289 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-19 07:56:29,289 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:29,289 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:29,289 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:29,289 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:29,289 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:29,289 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:29,309 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304199cc38301bf594494b897891411630950 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199cc38301bf594494b897891411630950 2023-04-19 07:56:29,315 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=193, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/fd4be2d75b834cd69f83da9710117f45 2023-04-19 07:56:29,326 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/1f165f673a4e4b6d8ab1d56f8b898b44 2023-04-19 07:56:29,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:29,331 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:29,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891049350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891049353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891049354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891049356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891049356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,366 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/b0242ec4007f404da580cb27ac27f26a 2023-04-19 07:56:29,370 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/fd4be2d75b834cd69f83da9710117f45 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/fd4be2d75b834cd69f83da9710117f45 2023-04-19 07:56:29,374 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/fd4be2d75b834cd69f83da9710117f45, entries=150, sequenceid=193, filesize=25.3 K 2023-04-19 07:56:29,375 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/1f165f673a4e4b6d8ab1d56f8b898b44 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1f165f673a4e4b6d8ab1d56f8b898b44 2023-04-19 07:56:29,379 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1f165f673a4e4b6d8ab1d56f8b898b44, entries=150, sequenceid=193, filesize=11.8 K 2023-04-19 07:56:29,379 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/b0242ec4007f404da580cb27ac27f26a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/b0242ec4007f404da580cb27ac27f26a 2023-04-19 07:56:29,382 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/b0242ec4007f404da580cb27ac27f26a, entries=150, sequenceid=193, filesize=11.8 K 2023-04-19 07:56:29,383 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=46.96 KB/48090 for 051c522ed965d63912b88bb381b9d1d2 in 94ms, sequenceid=193, compaction requested=true 2023-04-19 07:56:29,383 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:29,383 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:29,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:29,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:29,383 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:29,385 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,385 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,385 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,385 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dcf6372[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:29,387 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:29,387 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,388 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,388 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:29,388 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:29,388 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:29,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:29,390 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:29,390 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,390 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,390 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:29,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:29,393 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:29,393 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:29,393 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:29,393 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:29,393 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:29,396 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,396 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:29,397 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,398 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,399 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,402 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:29,402 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,402 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:29,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:29,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:29,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:29,402 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:29,403 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:29,403 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:29,402 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,403 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,403 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,403 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,403 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,403 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:29,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:29,455 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 07:56:29,455 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:29,455 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:29,455 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:29,455 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:29,455 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:29,455 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:29,469 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419f9619c20265f4516b83d3998b372eb79 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f9619c20265f4516b83d3998b372eb79 2023-04-19 07:56:29,480 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=204, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/14aac196c6694543a1417f6604ce4104 2023-04-19 07:56:29,503 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:29,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:29,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:29,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:29,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:29,506 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:29,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:29,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:29,506 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:29,507 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,507 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,508 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:29,508 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:29,508 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,508 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:29,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:29,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:29,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:29,510 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,510 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:29,510 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:29,510 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:29,512 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,512 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,512 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,512 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:29,513 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6714453f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:29,513 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,516 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,516 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:29,516 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:29,516 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:29,517 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,517 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:29,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890989501, max timestamp=1681890989516Num uniques 859; 2023-04-19 07:56:29,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890989501, max timestamp=1681890989516Num uniques 859; 2023-04-19 07:56:29,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:29,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890989501, max timestamp=1681890989516Num uniques 859; 2023-04-19 07:56:29,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:29,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:29,519 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:29,519 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:29,519 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,519 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:29,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:29,522 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:29,523 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:29,523 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:29,523 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:29,523 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:29,524 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,524 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,524 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:29,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,525 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,527 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,530 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,530 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,530 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:29,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:29,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:29,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:29,530 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:29,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:29,530 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:29,530 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,530 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,530 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,530 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:29,530 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:29,530 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,530 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:29,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891049587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,630 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:29,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:29,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:29,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:29,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:29,634 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:29,634 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:29,634 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:29,634 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:29,634 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,634 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,636 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:29,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:29,636 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:29,637 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,637 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:29,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:29,638 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:29,638 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:29,638 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:29,638 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:29,639 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:29,639 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,639 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:29,639 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:29,639 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,639 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:29,639 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:29,639 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,639 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:29,639 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:29,640 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:29,643 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@792c35af[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:29,643 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,644 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:29,645 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,645 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:29,646 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:29,646 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,646 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,646 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:29,649 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:29,649 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:29,649 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:29,649 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:29,649 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:29,660 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,660 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,660 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:29,661 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,663 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:29,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:29,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:29,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:29,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,666 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:29,666 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:29,666 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:29,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:29,666 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,667 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:29,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891049692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,766 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:29,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:29,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:29,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:29,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:29,770 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:29,770 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:29,770 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:29,770 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:29,770 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,771 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,772 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:29,772 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,772 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,772 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:29,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:29,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:29,773 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:29,773 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:29,773 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:29,774 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:29,774 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:29,774 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,774 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:29,774 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:29,774 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,774 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:29,774 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:29,774 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,774 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:29,774 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:29,776 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,776 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,777 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,777 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:29,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f38f632[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:29,778 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:29,778 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,779 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:29,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:29,780 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:29,780 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,780 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:29,781 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,781 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:29,781 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:29,782 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:29,786 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:29,786 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:29,786 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:29,786 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:29,786 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:29,788 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,788 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,788 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:29,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,789 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,790 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,794 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:29,794 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:29,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:29,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:29,794 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,794 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,794 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,794 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,794 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:29,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:29,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:29,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:29,795 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:29,795 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:29,794 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,795 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,795 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891049859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891049861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891049861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891049862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,890 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/171fa50402ff4afe88cb9f368cb531f1 2023-04-19 07:56:29,895 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:29,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:29,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891049896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:29,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:29,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:29,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:29,904 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:29,904 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:29,905 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:29,905 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:29,905 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,905 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,906 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,906 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:29,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:29,907 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,907 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:29,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:29,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:29,908 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:29,908 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:29,908 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:29,909 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:29,909 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:29,909 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,909 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:29,909 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:29,909 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:29,909 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:29,909 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:29,909 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,909 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:29,909 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:29,911 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,911 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:29,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69cac854[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:29,913 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:29,913 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,914 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,915 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:29,915 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:29,915 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:29,915 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,915 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:29,917 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,917 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:29,917 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:29,917 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:29,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:29,920 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:29,921 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:29,921 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:29,921 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:29,921 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:29,922 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:29,922 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,922 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:29,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:29,923 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:29,925 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:29,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:29,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:29,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:29,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:29,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:29,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:29,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:29,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:29,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,927 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:29,927 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:29,928 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:29,927 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:29,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:29,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:29,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:29,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:30,028 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:30,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:30,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:30,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:30,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:30,031 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:30,031 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:30,031 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:30,031 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:30,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,032 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,033 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:30,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:30,034 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,034 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,034 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:30,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:30,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:30,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:30,035 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:30,035 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:30,035 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:30,035 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:30,035 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:30,036 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:30,036 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,036 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:30,036 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:30,038 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,038 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:30,038 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,038 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:30,040 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c3255ef[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:30,040 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,041 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,042 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:30,042 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:30,042 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:30,042 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,042 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:30,043 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:30,043 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:30,043 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:30,044 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:30,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:30,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:30,046 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:30,046 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:30,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:30,047 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:30,047 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:30,048 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,048 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:30,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,048 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:30,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:30,049 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,051 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:30,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,056 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:30,056 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:30,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:30,057 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,057 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,057 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,057 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:30,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:30,057 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:30,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:30,057 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,057 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,057 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:30,057 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:30,058 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:30,158 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:30,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:30,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:30,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:30,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:30,162 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:30,162 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:30,162 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:30,162 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:30,162 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,162 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,164 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:30,164 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:30,164 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,164 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:30,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:30,165 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:30,165 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:30,165 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:30,166 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:30,166 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:30,166 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:30,166 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:30,166 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:30,166 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:30,166 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:30,166 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:30,166 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,166 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:30,166 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:30,168 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:30,168 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:30,168 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,168 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:30,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:30,170 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:30,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@342b9c8b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:30,170 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,177 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,177 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:30,177 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:30,177 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:30,177 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,177 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:30,179 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,179 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:30,179 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:30,179 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:30,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:30,181 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:30,181 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:30,181 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:30,181 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:30,181 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:30,183 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,183 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,183 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:30,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:30,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,184 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,185 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,187 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:30,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:30,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:30,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:30,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:30,188 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:30,188 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:30,187 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:30,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:30,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:30,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:30,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891050200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,288 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:30,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:30,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:30,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:30,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:30,291 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:30,291 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:30,291 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:30,291 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:30,292 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,292 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,297 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:30,297 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:30,298 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,298 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:30,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:30,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:30,299 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:30,299 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:30,299 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:30,299 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:30,299 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:30,299 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:30,299 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:30,299 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:30,300 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:30,300 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:30,300 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:30,300 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,300 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:30,300 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:30,301 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,301 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:30,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:30,304 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:30,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3caa655a[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:30,304 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,305 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:30,306 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,306 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:30,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,307 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:30,307 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:30,307 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:30,307 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/8a037f11aa4d47a89c7427116806156d 2023-04-19 07:56:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:30,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:30,312 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:30,312 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/14aac196c6694543a1417f6604ce4104 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/14aac196c6694543a1417f6604ce4104 2023-04-19 07:56:30,312 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:30,312 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:30,312 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:30,312 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:30,315 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,315 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,315 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:30,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:30,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/14aac196c6694543a1417f6604ce4104, entries=150, sequenceid=204, filesize=25.3 K 2023-04-19 07:56:30,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:30,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:30,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:30,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,316 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:30,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/171fa50402ff4afe88cb9f368cb531f1 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/171fa50402ff4afe88cb9f368cb531f1 2023-04-19 07:56:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,319 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:30,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/171fa50402ff4afe88cb9f368cb531f1, entries=150, sequenceid=204, filesize=11.8 K 2023-04-19 07:56:30,321 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/8a037f11aa4d47a89c7427116806156d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/8a037f11aa4d47a89c7427116806156d 2023-04-19 07:56:30,322 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,322 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:30,322 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,322 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,322 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,322 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:30,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:30,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:30,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:30,322 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:30,322 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:30,322 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:30,323 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:30,323 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:30,323 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:30,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:30,324 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/8a037f11aa4d47a89c7427116806156d, entries=150, sequenceid=204, filesize=11.8 K 2023-04-19 07:56:30,325 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 051c522ed965d63912b88bb381b9d1d2 in 870ms, sequenceid=204, compaction requested=true 2023-04-19 07:56:30,325 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:30,325 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:30,325 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:30,325 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:30,325 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:30,325 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:30,326 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:30,326 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/B is initiating minor compaction (all files) 2023-04-19 07:56:30,326 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/B in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:30,326 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/10cfcfd8fe4a4fd495ba7e94bdfa80df, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1f165f673a4e4b6d8ab1d56f8b898b44, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/171fa50402ff4afe88cb9f368cb531f1] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=35.5 K 2023-04-19 07:56:30,326 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77928 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:30,326 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/A is initiating minor compaction (all files) 2023-04-19 07:56:30,326 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/A in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:30,326 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/f60f9baca9dc444caf34f9dad2c83e8a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/fd4be2d75b834cd69f83da9710117f45, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/14aac196c6694543a1417f6604ce4104] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=76.1 K 2023-04-19 07:56:30,326 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 10cfcfd8fe4a4fd495ba7e94bdfa80df, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681890988649 2023-04-19 07:56:30,326 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting f60f9baca9dc444caf34f9dad2c83e8a, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681890988649 2023-04-19 07:56:30,327 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 1f165f673a4e4b6d8ab1d56f8b898b44, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681890988740 2023-04-19 07:56:30,327 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 171fa50402ff4afe88cb9f368cb531f1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681890989342 2023-04-19 07:56:30,327 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting fd4be2d75b834cd69f83da9710117f45, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681890988740 2023-04-19 07:56:30,327 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 14aac196c6694543a1417f6604ce4104, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681890989342 2023-04-19 07:56:30,332 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#B#compaction#527 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:30,336 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#A#compaction#528 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:30,357 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/fee620c5b44b49ff8fb011b2889dd985 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/fee620c5b44b49ff8fb011b2889dd985 2023-04-19 07:56:30,360 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/A of 051c522ed965d63912b88bb381b9d1d2 into fee620c5b44b49ff8fb011b2889dd985(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:30,360 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:30,360 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/A, priority=13, startTime=1681890990325; duration=0sec 2023-04-19 07:56:30,361 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:30,361 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:56:30,362 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48166 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:56:30,362 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/C is initiating minor compaction (all files) 2023-04-19 07:56:30,362 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/C in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:30,362 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/261e17d077c546358c96151c42782348, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/ca853285a6204f7997e3e2cbe8047737, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/b0242ec4007f404da580cb27ac27f26a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/8a037f11aa4d47a89c7427116806156d] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=47.0 K 2023-04-19 07:56:30,362 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 261e17d077c546358c96151c42782348, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681890987999 2023-04-19 07:56:30,362 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting ca853285a6204f7997e3e2cbe8047737, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681890988649 2023-04-19 07:56:30,363 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting b0242ec4007f404da580cb27ac27f26a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681890988740 2023-04-19 07:56:30,363 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 8a037f11aa4d47a89c7427116806156d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681890989342 2023-04-19 07:56:30,368 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#C#compaction#529 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:30,430 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:30,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:30,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:30,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:30,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:30,434 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:30,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:30,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:30,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:30,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,435 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:30,435 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:30,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:30,436 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,436 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:30,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:30,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:30,437 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:30,437 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:30,437 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:30,437 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:30,437 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:30,437 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:30,437 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:30,438 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-19 07:56:30,438 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:30,438 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:30,438 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:30,438 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:30,438 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:30,438 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:30,457 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041917e5ea676bc548d58583aee8bc0e6145 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041917e5ea676bc548d58583aee8bc0e6145 2023-04-19 07:56:30,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:30,710 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:30,740 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/9c4f36467a944fd8bd565d625f1313e3 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9c4f36467a944fd8bd565d625f1313e3 2023-04-19 07:56:30,744 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/B of 051c522ed965d63912b88bb381b9d1d2 into 9c4f36467a944fd8bd565d625f1313e3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:30,744 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:30,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:30,744 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/B, priority=13, startTime=1681890990325; duration=0sec 2023-04-19 07:56:30,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891050742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,745 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:30,778 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/e2d18907838a4f9d87b5bb0fb02fac35 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e2d18907838a4f9d87b5bb0fb02fac35 2023-04-19 07:56:30,782 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/C of 051c522ed965d63912b88bb381b9d1d2 into e2d18907838a4f9d87b5bb0fb02fac35(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:30,782 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:30,782 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/C, priority=12, startTime=1681890990325; duration=0sec 2023-04-19 07:56:30,782 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:30,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:30,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891050846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,862 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=241, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/a8ce5fc26b7a454da77cb4054b09d1a8 2023-04-19 07:56:30,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:30,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891050863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:30,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891050867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:30,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891050867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:30,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:30,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891050868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:31,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891051047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,275 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/10602b28edb94137bb77b571a6becad1 2023-04-19 07:56:31,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:31,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37938 deadline: 1681891051352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,707 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/0839ad88c826439fb918cd40f11d62fb 2023-04-19 07:56:31,711 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/a8ce5fc26b7a454da77cb4054b09d1a8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/a8ce5fc26b7a454da77cb4054b09d1a8 2023-04-19 07:56:31,715 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/a8ce5fc26b7a454da77cb4054b09d1a8, entries=150, sequenceid=241, filesize=25.3 K 2023-04-19 07:56:31,716 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/10602b28edb94137bb77b571a6becad1 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/10602b28edb94137bb77b571a6becad1 2023-04-19 07:56:31,719 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/10602b28edb94137bb77b571a6becad1, entries=150, sequenceid=241, filesize=11.8 K 2023-04-19 07:56:31,719 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/0839ad88c826439fb918cd40f11d62fb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/0839ad88c826439fb918cd40f11d62fb 2023-04-19 07:56:31,723 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/0839ad88c826439fb918cd40f11d62fb, entries=150, sequenceid=241, filesize=11.8 K 2023-04-19 07:56:31,724 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 051c522ed965d63912b88bb381b9d1d2 in 1287ms, sequenceid=241, compaction requested=false 2023-04-19 07:56:31,724 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:31,724 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:31,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:31,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:31,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:31,724 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:31,726 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:31,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:31,726 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:31,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:31,727 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:31,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39d92f3[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:31,727 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:31,729 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:31,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:31,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:31,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:31,729 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,729 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:31,729 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:31,730 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:31,731 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:31,731 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:31,731 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:31,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:31,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:31,734 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:31,734 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:31,734 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:31,734 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:31,734 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:31,736 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:31,736 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:31,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:31,736 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:31,737 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:31,738 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:31,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,744 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,744 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:31,744 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:31,744 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,744 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:31,744 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:31,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:31,744 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:31,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:31,744 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:31,744 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:31,744 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:31,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:31,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:31,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:31,745 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:31,745 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:31,845 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:31,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:31,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:31,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:31,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:31,848 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:31,848 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:31,848 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:31,848 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:31,849 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:31,849 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:31,850 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,850 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:31,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:31,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:31,851 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:31,851 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:31,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:31,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:31,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:31,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:31,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:31,852 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:31,852 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:31,853 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:31,853 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 07:56:31,853 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:31,853 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:31,853 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:31,853 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:31,853 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:31,853 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:31,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:31,868 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:31,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890991902, max timestamp=1681890991916Num uniques 859; 2023-04-19 07:56:31,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890991902, max timestamp=1681890991916Num uniques 859; 2023-04-19 07:56:31,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:31,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890991902, max timestamp=1681890991916Num uniques 859; 2023-04-19 07:56:31,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:31,918 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:31,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:31,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891051957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:32,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:32,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891052060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:32,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:32,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891052264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:32,267 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419b8791152d2354d9092d0da98d9af8592 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b8791152d2354d9092d0da98d9af8592 2023-04-19 07:56:32,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:32,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891052568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:32,672 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=254, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/2bd5ae90fc044556a55a3ab2b8ed41da 2023-04-19 07:56:32,683 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/1b547ada454347e5bea848cea2f0744e 2023-04-19 07:56:32,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:32,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891052870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:32,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:32,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891052871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:32,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:32,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891052871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:32,874 DEBUG [Thread-3716] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:56:32,875 DEBUG [Thread-3714] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4133 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:56:32,876 DEBUG [Thread-3712] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:56:32,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:32,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891052885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:32,889 DEBUG [Thread-3708] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:56:33,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:33,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891053080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,093 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/8eb76521429f42e5b8dfad0226af6ae3 2023-04-19 07:56:33,097 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/2bd5ae90fc044556a55a3ab2b8ed41da as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/2bd5ae90fc044556a55a3ab2b8ed41da 2023-04-19 07:56:33,100 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/2bd5ae90fc044556a55a3ab2b8ed41da, entries=100, sequenceid=254, filesize=18.5 K 2023-04-19 07:56:33,101 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/1b547ada454347e5bea848cea2f0744e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1b547ada454347e5bea848cea2f0744e 2023-04-19 07:56:33,104 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1b547ada454347e5bea848cea2f0744e, entries=100, sequenceid=254, filesize=9.4 K 2023-04-19 07:56:33,105 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/8eb76521429f42e5b8dfad0226af6ae3 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/8eb76521429f42e5b8dfad0226af6ae3 2023-04-19 07:56:33,107 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/8eb76521429f42e5b8dfad0226af6ae3, entries=100, sequenceid=254, filesize=9.4 K 2023-04-19 07:56:33,108 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 051c522ed965d63912b88bb381b9d1d2 in 1255ms, sequenceid=254, compaction requested=true 2023-04-19 07:56:33,108 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:33,108 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:33,108 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:33,108 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:33,108 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,109 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:33,109 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:33,111 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,111 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,111 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,111 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:33,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dbe2242[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:33,113 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:33,113 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,114 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:33,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:33,114 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:33,114 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,114 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:33,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:33,116 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,116 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:33,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:33,119 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:33,119 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:33,119 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:33,119 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:33,119 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:33,121 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,121 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,121 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:33,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:33,122 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,123 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,126 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,126 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,126 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,126 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,126 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,126 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:33,126 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,126 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:33,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:33,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:33,126 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:33,126 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:33,126 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:33,126 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:33,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:33,226 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:33,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:33,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:33,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:33,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:33,230 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:33,230 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:33,230 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:33,230 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:33,230 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,230 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,232 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:33,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:33,232 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,232 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,232 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:33,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:33,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:33,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:33,234 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:33,234 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:33,234 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:33,234 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:33,234 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-19 07:56:33,234 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:33,234 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:33,234 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:33,234 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:33,234 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:33,234 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:33,250 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041904fdaef7d0324b97aeb4afb1e27023aa to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041904fdaef7d0324b97aeb4afb1e27023aa 2023-04-19 07:56:33,255 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=290, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/67f11c86567f41d2ae4290566fa6b90f 2023-04-19 07:56:33,264 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/ee1faafe31a74cee939012d80445f521 2023-04-19 07:56:33,273 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/b1eba3a5f5bc4fe1bcbf20d3fa522694 2023-04-19 07:56:33,277 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/67f11c86567f41d2ae4290566fa6b90f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/67f11c86567f41d2ae4290566fa6b90f 2023-04-19 07:56:33,280 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/67f11c86567f41d2ae4290566fa6b90f, entries=150, sequenceid=290, filesize=25.4 K 2023-04-19 07:56:33,281 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/ee1faafe31a74cee939012d80445f521 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ee1faafe31a74cee939012d80445f521 2023-04-19 07:56:33,284 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ee1faafe31a74cee939012d80445f521, entries=150, sequenceid=290, filesize=11.9 K 2023-04-19 07:56:33,284 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/b1eba3a5f5bc4fe1bcbf20d3fa522694 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/b1eba3a5f5bc4fe1bcbf20d3fa522694 2023-04-19 07:56:33,288 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/b1eba3a5f5bc4fe1bcbf20d3fa522694, entries=150, sequenceid=290, filesize=11.9 K 2023-04-19 07:56:33,289 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 051c522ed965d63912b88bb381b9d1d2 in 55ms, sequenceid=290, compaction requested=true 2023-04-19 07:56:33,289 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:33,289 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:33,289 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:33,289 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:33,289 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,289 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:33,289 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:33,292 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,292 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,293 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,293 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:33,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55993eba[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:33,295 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:33,295 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,296 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:33,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:33,296 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,296 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:33,296 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:33,298 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:33,298 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:33,298 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:33,298 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:33,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:33,300 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:33,300 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:33,300 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:33,301 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:33,301 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:33,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,302 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:33,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:33,303 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,304 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:33,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:33,307 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:33,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:33,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:33,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:33,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:33,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:33,307 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:33,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:33,308 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:33,308 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:33,408 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:33,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:33,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:33,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:33,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:33,412 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:33,412 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:33,412 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:33,412 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:33,412 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,412 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,413 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:33,413 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:33,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:33,414 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,414 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:33,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:33,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:33,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:33,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:33,415 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:33,415 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:33,415 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:33,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:33,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:33,416 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:33,416 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:33,416 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:33,416 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:33,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:33,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:33,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:33,416 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:33,418 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,418 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:33,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:33,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3af601e6[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:33,420 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:33,420 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,421 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,421 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:33,421 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:33,422 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:33,422 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,422 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:33,424 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,424 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:33,424 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:33,424 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:33,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:33,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:33,427 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:33,427 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:33,427 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:33,427 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:33,427 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:33,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,428 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:33,429 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,430 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,433 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:33,433 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:33,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:33,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:33,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:33,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:33,433 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,433 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:33,433 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,433 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,433 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,433 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:33,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:33,433 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:33,433 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:33,433 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,433 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,533 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:33,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:33,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:33,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:33,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:33,537 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:33,537 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:33,537 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:33,537 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:33,538 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,538 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,539 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,539 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:33,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:33,539 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,539 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:33,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:33,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:33,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:33,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:33,540 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:33,541 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:33,541 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:33,541 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:33,541 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:33,541 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:33,541 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:33,541 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:33,541 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:33,541 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:33,541 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:33,541 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,541 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:33,541 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:33,547 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,547 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:33,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:33,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55131641[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:33,558 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:33,558 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,559 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,559 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:33,559 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,559 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:33,559 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:33,559 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:33,561 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,561 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:33,561 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:33,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,561 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:33,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:33,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:33,564 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:33,564 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:33,564 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:33,564 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:33,564 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:33,565 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,566 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,566 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:33,566 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,568 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:33,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:33,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,572 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:33,572 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:33,572 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,576 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:33,576 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:33,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:33,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:33,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:33,673 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:33,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:33,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:33,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:33,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:33,676 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:33,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:33,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:33,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:33,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,679 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:33,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:33,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:33,679 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,679 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:33,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:33,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:33,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:33,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:33,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:33,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:33,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:33,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:33,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:33,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:33,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:33,681 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:33,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:33,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:33,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:33,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:33,683 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,683 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,683 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,683 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:33,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:33,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19f38293[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:33,685 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:33,685 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,686 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,686 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:33,686 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:33,686 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:33,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:33,693 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:33,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,693 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:33,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:33,696 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:33,696 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:33,696 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:33,696 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:33,696 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:33,698 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,698 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:33,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,698 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,699 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,702 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:33,702 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,702 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:33,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,703 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:33,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:33,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:33,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:33,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:33,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,703 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:33,703 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:33,703 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:33,803 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:33,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:33,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:33,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:33,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:33,807 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:33,807 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:33,807 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:33,807 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:33,808 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,808 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,809 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:33,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:33,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:33,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:33,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:33,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:33,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:33,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:33,811 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:33,811 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:33,811 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:33,811 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:33,811 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:33,812 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:33,812 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:33,812 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:33,812 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:33,812 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:33,812 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:33,812 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,812 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:33,812 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:33,815 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,815 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,815 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:33,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50303f9[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:33,817 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:33,817 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,819 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:33,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:33,819 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,819 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:33,819 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:33,821 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:33,821 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,821 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,821 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:33,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:33,824 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:33,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:33,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:33,824 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:33,824 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:33,825 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,825 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,826 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,826 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,827 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,830 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:33,830 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,830 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:33,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:33,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:33,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:33,830 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:33,830 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:33,830 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:33,831 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:33,830 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,831 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,831 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,831 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,831 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:33,931 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:33,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:33,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:33,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:33,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:33,934 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:33,934 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:33,934 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:33,934 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:33,935 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,935 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,936 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:33,936 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:33,936 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,936 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:33,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:33,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:33,938 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:33,938 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:33,939 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:33,939 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:33,940 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,940 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,941 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,941 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:33,942 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:33,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ff0db8e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:33,942 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,947 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:33,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:33,947 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:33,947 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,947 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:33,948 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,949 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:33,951 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:33,951 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:33,951 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:33,951 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:33,951 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:33,953 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,953 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:33,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,953 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:33,953 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,955 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,957 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:33,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:33,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:33,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:33,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:33,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:33,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:33,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:33,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:33,957 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:33,957 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:33,958 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:33,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:33,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,058 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:34,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:34,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:34,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:34,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:34,061 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:34,061 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,061 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:34,061 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:34,062 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,062 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,064 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,064 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,064 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,064 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:34,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:34,066 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:34,066 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,066 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:34,066 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:34,066 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:34,066 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,066 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:34,066 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:34,066 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,066 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:34,066 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:34,066 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,066 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:34,067 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:34,068 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:34,070 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:34,070 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@429c752f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:34,071 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,071 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:34,071 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:34,071 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:34,071 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,071 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:34,072 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:34,073 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,073 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,073 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:34,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:34,075 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:34,075 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:34,075 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:34,075 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:34,075 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:34,078 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,078 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,078 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,079 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,081 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,083 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,083 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:34,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,083 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,084 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,084 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,084 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:34,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,084 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:34,084 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,084 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,084 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,084 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:34,139 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:56:34,140 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:34,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:34,140 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:34,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:34,140 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:34,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:34,154 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304195889e10f3b264b5a863d2a2b307a029c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195889e10f3b264b5a863d2a2b307a029c 2023-04-19 07:56:34,184 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:34,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:34,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:34,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:34,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:34,187 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:34,187 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,187 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:34,187 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:34,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,189 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,189 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,189 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,189 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:34,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:34,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:34,191 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:34,191 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,191 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:34,196 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,196 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:34,196 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:34,196 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:34,198 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,198 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,198 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,198 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47fe80f5[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:34,200 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:34,200 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,202 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,202 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:34,202 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:34,202 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:34,203 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,203 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:34,204 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:34,204 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,204 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:34,204 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:34,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:34,207 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:34,207 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:34,207 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:34,207 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:34,207 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:34,209 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,209 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,209 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,210 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,212 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,214 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,214 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,214 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,214 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,214 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,214 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,214 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,214 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,214 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,214 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:34,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:34,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:34,215 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:34,215 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:34,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890994195, max timestamp=1681890994215Num uniques 859; 2023-04-19 07:56:34,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890994195, max timestamp=1681890994215Num uniques 859; 2023-04-19 07:56:34,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:34,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:34,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890994195, max timestamp=1681890994215Num uniques 859; 2023-04-19 07:56:34,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:34,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:34,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891054279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,315 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:34,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:34,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:34,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:34,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:34,319 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:34,319 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,319 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:34,319 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:34,319 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,319 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,322 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,322 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,322 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,322 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:34,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:34,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:34,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:34,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:34,325 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:34,325 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,325 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:34,325 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:34,325 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:34,325 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,325 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:34,325 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:34,325 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,325 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:34,326 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:34,326 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,326 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:34,326 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:34,331 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,331 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,331 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,331 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:34,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a7f4f38[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:34,333 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:34,333 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,334 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,334 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:34,335 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:34,335 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,335 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:34,335 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:34,336 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,336 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:34,336 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,336 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:34,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:34,339 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:34,339 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:34,339 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:34,339 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:34,339 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:34,344 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,344 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,344 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,345 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,346 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,349 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,349 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,349 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,349 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,349 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:34,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:34,349 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,350 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,350 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,350 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:34,350 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:34,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:34,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891054384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,450 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:34,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:34,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:34,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:34,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:34,453 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:34,453 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,453 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:34,453 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:34,454 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,454 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,455 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,455 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,456 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,456 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:34,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:34,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:34,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:34,457 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:34,457 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,457 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:34,457 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:34,457 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:34,457 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,458 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:34,458 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:34,458 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,458 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:34,458 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:34,458 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,458 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:34,458 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:34,459 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,459 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,460 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,460 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:34,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@535ac352[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:34,461 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:34,461 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,464 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,464 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:34,464 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:34,464 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:34,464 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,465 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:34,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:34,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,466 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,466 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:34,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:34,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:34,468 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:34,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:34,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:34,469 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:34,469 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:34,470 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,470 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,470 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,471 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,472 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,474 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,474 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,474 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,474 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:34,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:34,474 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,475 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,475 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,475 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,475 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,475 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,475 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:34,475 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:34,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:34,574 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=306, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/56bf700403aa4dd3ba5c885150e3a4a8 2023-04-19 07:56:34,575 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:34,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:34,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:34,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:34,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:34,579 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:34,579 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,579 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:34,579 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:34,579 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,579 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,582 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:34,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:34,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:34,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:34,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:34,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:34,584 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:34,584 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:34,584 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,584 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:34,584 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:34,584 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,584 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:34,584 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:34,584 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,584 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:34,584 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:34,586 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,586 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,587 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,587 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:34,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72cef11b[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:34,588 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:34,588 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,589 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,589 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:34,589 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:34,589 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:34,589 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,590 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:34,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:34,591 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:34,591 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,591 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891054589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,591 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:34,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:34,593 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:34,593 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:34,593 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:34,593 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:34,593 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:34,595 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,595 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,596 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,596 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,598 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,600 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,600 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,600 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,600 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,600 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,600 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:34,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:34,600 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,600 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,600 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:34,600 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,600 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:34,600 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,700 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:34,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:34,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:34,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:34,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:34,704 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:34,704 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,704 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:34,704 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:34,705 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,705 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,706 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:34,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:34,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:34,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:34,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:34,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:34,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:34,708 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,708 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:34,708 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:34,708 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:34,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:34,708 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,709 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:34,709 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:34,710 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,711 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,711 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:34,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41043e9f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:34,712 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:34,712 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,715 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,715 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:34,715 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:34,715 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:34,715 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,715 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:34,716 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:34,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:34,719 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:34,719 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:34,720 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:34,720 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:34,720 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:34,721 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,721 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,721 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,722 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,723 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,733 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,733 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,733 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,733 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:34,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:34,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:34,733 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,733 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,733 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,733 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,733 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,734 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,733 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:34,734 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,834 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:34,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:34,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:34,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:34,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:34,837 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:34,837 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,837 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:34,837 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:34,838 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,838 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,839 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,839 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,840 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,840 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:34,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:34,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:34,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:34,841 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:34,841 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,841 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:34,841 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:34,841 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:34,841 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,841 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:34,841 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:34,842 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:34,842 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:34,843 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,844 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:34,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61a2af08[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:34,845 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:34,845 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,846 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,847 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:34,847 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:34,847 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:34,847 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,847 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:34,849 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,849 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:34,849 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,849 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:34,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:34,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:34,853 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:34,853 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:34,854 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:34,854 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:34,854 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:34,855 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,855 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,855 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,856 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,857 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,859 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:34,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:34,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,859 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,860 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,860 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,860 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:34,860 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:34,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:34,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:34,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891054892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,960 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:34,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:34,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:34,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:34,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:34,964 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:34,964 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,964 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:34,964 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:34,965 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,965 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,966 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,966 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,966 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,966 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:34,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:34,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:34,968 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:34,968 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:34,968 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:34,968 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:34,968 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,968 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:34,968 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:34,968 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:34,968 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:34,968 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:34,968 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,968 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:34,968 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,971 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:34,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6467b03d[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:34,973 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:34,973 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,974 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,974 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:34,974 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:34,974 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:34,975 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,975 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:34,975 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,976 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:34,976 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,976 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:34,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:34,978 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:34,978 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:34,978 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:34,978 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:34,978 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:34,980 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,980 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:34,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,980 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:34,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,980 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,982 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,985 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,985 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,985 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:34,985 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:34,985 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,985 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:34,985 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:34,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:34,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:34,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:34,985 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:34,985 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:34,985 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:34,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:34,986 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:34,986 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:34,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:34,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/a715d6c142b842c7a6540fab5bcb378c 2023-04-19 07:56:34,999 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/2959a99a08e44ac190914fd373fd5b59 2023-04-19 07:56:35,003 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/56bf700403aa4dd3ba5c885150e3a4a8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/56bf700403aa4dd3ba5c885150e3a4a8 2023-04-19 07:56:35,007 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/56bf700403aa4dd3ba5c885150e3a4a8, entries=200, sequenceid=306, filesize=32.3 K 2023-04-19 07:56:35,008 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/a715d6c142b842c7a6540fab5bcb378c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/a715d6c142b842c7a6540fab5bcb378c 2023-04-19 07:56:35,011 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/a715d6c142b842c7a6540fab5bcb378c, entries=150, sequenceid=306, filesize=11.9 K 2023-04-19 07:56:35,012 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/2959a99a08e44ac190914fd373fd5b59 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/2959a99a08e44ac190914fd373fd5b59 2023-04-19 07:56:35,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/2959a99a08e44ac190914fd373fd5b59, entries=150, sequenceid=306, filesize=11.9 K 2023-04-19 07:56:35,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 051c522ed965d63912b88bb381b9d1d2 in 877ms, sequenceid=306, compaction requested=true 2023-04-19 07:56:35,016 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:35,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:35,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:35,016 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:56:35,016 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:56:35,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:35,017 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130164 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:56:35,017 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/A is initiating minor compaction (all files) 2023-04-19 07:56:35,017 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58428 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:56:35,017 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/A in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:35,017 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/B is initiating minor compaction (all files) 2023-04-19 07:56:35,017 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/fee620c5b44b49ff8fb011b2889dd985, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/a8ce5fc26b7a454da77cb4054b09d1a8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/2bd5ae90fc044556a55a3ab2b8ed41da, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/67f11c86567f41d2ae4290566fa6b90f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/56bf700403aa4dd3ba5c885150e3a4a8] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=127.1 K 2023-04-19 07:56:35,017 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/B in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:35,018 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9c4f36467a944fd8bd565d625f1313e3, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/10602b28edb94137bb77b571a6becad1, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1b547ada454347e5bea848cea2f0744e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ee1faafe31a74cee939012d80445f521, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/a715d6c142b842c7a6540fab5bcb378c] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=57.1 K 2023-04-19 07:56:35,018 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting fee620c5b44b49ff8fb011b2889dd985, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681890989342 2023-04-19 07:56:35,018 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 9c4f36467a944fd8bd565d625f1313e3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681890989342 2023-04-19 07:56:35,018 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting a8ce5fc26b7a454da77cb4054b09d1a8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1681890989577 2023-04-19 07:56:35,018 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 10602b28edb94137bb77b571a6becad1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1681890989577 2023-04-19 07:56:35,018 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 2bd5ae90fc044556a55a3ab2b8ed41da, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681890990735 2023-04-19 07:56:35,019 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 1b547ada454347e5bea848cea2f0744e, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681890990735 2023-04-19 07:56:35,019 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 67f11c86567f41d2ae4290566fa6b90f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1681890991944 2023-04-19 07:56:35,019 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting ee1faafe31a74cee939012d80445f521, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1681890991944 2023-04-19 07:56:35,019 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 56bf700403aa4dd3ba5c885150e3a4a8, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681890994116 2023-04-19 07:56:35,020 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting a715d6c142b842c7a6540fab5bcb378c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681890994116 2023-04-19 07:56:35,027 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#B#compaction#542 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:35,043 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#A#compaction#543 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:35,086 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:35,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:35,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:35,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:35,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:35,089 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:35,089 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:35,089 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:35,089 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:35,090 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:35,090 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:35,091 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:35,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:35,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:35,091 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:35,092 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:35,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:35,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:35,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:35,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:35,093 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:35,093 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:35,093 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:35,093 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:35,093 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:35,093 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:35,093 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:35,093 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-19 07:56:35,094 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:35,094 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:35,094 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:35,094 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:35,094 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:35,094 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:35,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:35,401 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:35,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:35,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891055433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:35,438 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/db55488ce52f487397deb8c61da7b24e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/db55488ce52f487397deb8c61da7b24e 2023-04-19 07:56:35,442 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/B of 051c522ed965d63912b88bb381b9d1d2 into db55488ce52f487397deb8c61da7b24e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:35,443 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:35,443 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/B, priority=11, startTime=1681890995016; duration=0sec 2023-04-19 07:56:35,443 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:35,443 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-19 07:56:35,444 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58394 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-19 07:56:35,444 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/C is initiating minor compaction (all files) 2023-04-19 07:56:35,444 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/C in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:35,445 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e2d18907838a4f9d87b5bb0fb02fac35, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/0839ad88c826439fb918cd40f11d62fb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/8eb76521429f42e5b8dfad0226af6ae3, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/b1eba3a5f5bc4fe1bcbf20d3fa522694, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/2959a99a08e44ac190914fd373fd5b59] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=57.0 K 2023-04-19 07:56:35,445 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting e2d18907838a4f9d87b5bb0fb02fac35, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681890989342 2023-04-19 07:56:35,445 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 0839ad88c826439fb918cd40f11d62fb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1681890989577 2023-04-19 07:56:35,446 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 8eb76521429f42e5b8dfad0226af6ae3, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681890990735 2023-04-19 07:56:35,446 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting b1eba3a5f5bc4fe1bcbf20d3fa522694, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1681890991944 2023-04-19 07:56:35,446 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 2959a99a08e44ac190914fd373fd5b59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681890994116 2023-04-19 07:56:35,460 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#C#compaction#545 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:35,513 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304197d921f971ed7496ea7d458e2769de2fb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197d921f971ed7496ea7d458e2769de2fb 2023-04-19 07:56:35,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:35,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891055539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:35,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:35,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891055743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:35,880 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/a1692cdd1e0e4fbcbe338066db255461 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/a1692cdd1e0e4fbcbe338066db255461 2023-04-19 07:56:35,880 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/98d84c6d1bd7466686f7a928e3b2e7e9 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/98d84c6d1bd7466686f7a928e3b2e7e9 2023-04-19 07:56:35,884 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/A of 051c522ed965d63912b88bb381b9d1d2 into 98d84c6d1bd7466686f7a928e3b2e7e9(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:35,884 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/C of 051c522ed965d63912b88bb381b9d1d2 into a1692cdd1e0e4fbcbe338066db255461(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:35,884 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:35,884 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/C, priority=11, startTime=1681890995016; duration=0sec 2023-04-19 07:56:35,885 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:35,884 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:35,885 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/A, priority=11, startTime=1681890995016; duration=0sec 2023-04-19 07:56:35,885 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:35,918 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=343, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/3a2baedd34c743bcba794599f0d49e6c 2023-04-19 07:56:35,928 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/262ecd5001f34e07aa42f72fa646a464 2023-04-19 07:56:36,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:36,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891056050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,337 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/cf473e537bad420598b81c058c48d831 2023-04-19 07:56:36,341 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/3a2baedd34c743bcba794599f0d49e6c as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/3a2baedd34c743bcba794599f0d49e6c 2023-04-19 07:56:36,345 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/3a2baedd34c743bcba794599f0d49e6c, entries=150, sequenceid=343, filesize=25.4 K 2023-04-19 07:56:36,345 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/262ecd5001f34e07aa42f72fa646a464 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/262ecd5001f34e07aa42f72fa646a464 2023-04-19 07:56:36,350 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/262ecd5001f34e07aa42f72fa646a464, entries=150, sequenceid=343, filesize=11.9 K 2023-04-19 07:56:36,351 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/cf473e537bad420598b81c058c48d831 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/cf473e537bad420598b81c058c48d831 2023-04-19 07:56:36,354 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/cf473e537bad420598b81c058c48d831, entries=150, sequenceid=343, filesize=11.9 K 2023-04-19 07:56:36,355 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 051c522ed965d63912b88bb381b9d1d2 in 1262ms, sequenceid=343, compaction requested=false 2023-04-19 07:56:36,355 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:36,355 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:36,355 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:36,355 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:36,355 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,355 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:36,356 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:36,357 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,357 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,358 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,358 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:36,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61a4c383[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:36,360 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:36,360 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,361 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,361 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:36,361 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:36,361 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:36,361 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,362 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:36,362 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:36,363 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:36,363 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:36,363 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:36,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:36,366 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:36,366 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:36,366 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:36,366 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:36,366 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:36,368 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,368 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:36,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,369 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:36,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:36,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,370 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,371 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,377 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,377 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,377 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,377 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:36,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:36,377 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:36,377 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:36,377 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:36,377 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:36,377 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,378 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:36,378 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:36,378 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:36,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:36,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:36,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:36,478 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:36,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:36,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:36,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:36,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:36,481 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:36,481 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:36,482 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:36,482 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:36,482 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,482 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,483 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:36,483 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,484 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,484 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:36,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:36,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:36,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:36,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:36,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:36,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:36,485 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:36,485 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:36,485 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:36,486 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:56:36,486 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:36,486 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:36,486 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:36,486 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:36,486 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:36,486 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:36,505 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304192e35ef8b31c44a32ae1448c608bddef3 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192e35ef8b31c44a32ae1448c608bddef3 2023-04-19 07:56:36,510 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=356, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/d91851fd4011453fb0ecbd6b39533abf 2023-04-19 07:56:36,527 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/e1458116ab42486fa82f77bf444ce7c5 2023-04-19 07:56:36,538 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/da445a01280d475fafd06b7d1ca4ff74 2023-04-19 07:56:36,542 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/d91851fd4011453fb0ecbd6b39533abf as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/d91851fd4011453fb0ecbd6b39533abf 2023-04-19 07:56:36,545 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/d91851fd4011453fb0ecbd6b39533abf, entries=150, sequenceid=356, filesize=25.4 K 2023-04-19 07:56:36,546 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/e1458116ab42486fa82f77bf444ce7c5 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/e1458116ab42486fa82f77bf444ce7c5 2023-04-19 07:56:36,549 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/e1458116ab42486fa82f77bf444ce7c5, entries=150, sequenceid=356, filesize=11.9 K 2023-04-19 07:56:36,550 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/da445a01280d475fafd06b7d1ca4ff74 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/da445a01280d475fafd06b7d1ca4ff74 2023-04-19 07:56:36,553 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/da445a01280d475fafd06b7d1ca4ff74, entries=150, sequenceid=356, filesize=11.9 K 2023-04-19 07:56:36,554 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 051c522ed965d63912b88bb381b9d1d2 in 69ms, sequenceid=356, compaction requested=true 2023-04-19 07:56:36,554 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:36,554 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:36,554 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:36,554 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:36,554 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,554 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:36,554 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:36,557 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,557 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,558 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:36,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b4f9ee[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:36,559 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:36,559 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,561 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,561 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:36,561 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:36,561 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:36,561 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,561 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:36,562 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,562 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:36,562 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:36,562 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:36,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:36,565 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:36,565 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:36,565 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:36,565 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:36,565 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:36,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,567 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:36,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:36,567 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,569 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:36,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:36,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,572 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:36,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:36,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:36,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:36,572 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:36,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:36,572 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:36,573 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:36,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:36,608 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:56:36,608 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:36,608 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:36,608 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:36,608 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:36,608 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:36,608 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:36,620 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304199f5c1dd497ad415aafc1611c0913f09a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199f5c1dd497ad415aafc1611c0913f09a 2023-04-19 07:56:36,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890996651, max timestamp=1681890996667Num uniques 859; 2023-04-19 07:56:36,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:36,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890996651, max timestamp=1681890996667Num uniques 859; 2023-04-19 07:56:36,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890996651, max timestamp=1681890996667Num uniques 859; 2023-04-19 07:56:36,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:36,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:36,673 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:36,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:36,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:36,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:36,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:36,676 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:36,676 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:36,676 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:36,676 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:36,676 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,676 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,677 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,677 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:36,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:36,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:36,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,678 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,678 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:36,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:36,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:36,679 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:36,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:36,679 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:36,680 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:36,680 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:36,680 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:36,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:36,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:36,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:36,680 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:36,681 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,681 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,682 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,682 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:36,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@751adc3f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:36,683 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:36,683 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,684 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,684 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:36,685 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:36,685 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:36,685 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,685 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:36,687 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,687 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,687 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:36,687 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:36,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:36,690 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:36,690 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:36,690 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:36,690 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:36,690 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:36,696 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,696 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,696 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:36,697 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,698 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:36,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:36,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:36,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:36,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:36,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:36,700 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:36,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:36,700 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:36,700 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:36,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:36,700 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,701 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,701 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:36,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891056733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,801 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:36,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:36,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:36,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:36,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:36,804 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:36,804 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:36,804 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:36,804 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:36,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,806 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:36,806 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:36,806 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,806 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:36,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:36,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:36,808 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:36,808 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:36,808 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:36,808 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:36,809 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:36,809 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:36,809 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:36,809 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:36,809 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:36,809 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:36,809 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:36,809 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,809 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:36,809 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:36,811 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,811 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,811 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,812 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:36,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c257cf9[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:36,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:36,813 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,814 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,814 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:36,814 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:36,814 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:36,814 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,815 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:36,816 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:36,816 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:36,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:36,819 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:36,819 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:36,819 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:36,819 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:36,819 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:36,821 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,821 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,821 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:36,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,821 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,822 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,825 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,825 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,825 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,825 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,826 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,826 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,826 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:36,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:36,826 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:36,826 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:36,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:36,826 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:36,826 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:36,826 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:36,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:36,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:36,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:36,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:36,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891056836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:36,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37884 deadline: 1681891056890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,893 DEBUG [Thread-3714] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:56:36,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:36,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37914 deadline: 1681891056891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,896 DEBUG [Thread-3708] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:56:36,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:36,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37930 deadline: 1681891056898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,901 DEBUG [Thread-3712] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:56:36,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:36,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37898 deadline: 1681891056911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,913 DEBUG [Thread-3716] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., hostname=jenkins-hbase4.apache.org,42359,1681890819191, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-19 07:56:36,926 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:36,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:36,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:36,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:36,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:36,930 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:36,930 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:36,930 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:36,930 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:36,931 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,931 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,932 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:36,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:36,932 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:36,932 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,933 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:36,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:36,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:36,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:36,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:36,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:36,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:36,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:36,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:36,934 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:36,934 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:36,934 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:36,934 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:36,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:36,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:36,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,935 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:36,935 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:36,939 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,939 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,939 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,939 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:36,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bda1a25[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:36,940 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:36,940 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,942 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,943 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:36,943 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:36,943 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:36,943 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,943 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:36,944 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:36,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,944 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:36,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:36,947 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:36,947 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:36,947 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:36,947 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:36,947 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:36,949 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,949 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:36,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:36,949 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:36,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:36,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:36,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:36,950 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:36,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,952 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,954 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,954 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,954 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:36,954 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:36,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:36,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:36,955 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:36,955 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:36,955 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:36,955 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:36,954 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:36,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:36,955 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:36,955 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:36,955 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:36,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:36,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:37,025 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=366, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/6cdb3ee83c954680b0cd45886a6b0d6a 2023-04-19 07:56:37,034 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/7ef114e792634d5684947866c4594c4e 2023-04-19 07:56:37,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:37,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891057040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/5750e6e7ddd849d8ba1655503427b5c0 2023-04-19 07:56:37,055 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:37,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:37,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:37,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/6cdb3ee83c954680b0cd45886a6b0d6a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/6cdb3ee83c954680b0cd45886a6b0d6a 2023-04-19 07:56:37,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:37,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:37,059 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:37,059 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:37,059 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:37,059 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:37,059 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,060 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,062 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:37,062 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/6cdb3ee83c954680b0cd45886a6b0d6a, entries=150, sequenceid=366, filesize=25.4 K 2023-04-19 07:56:37,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:37,062 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,062 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,063 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/7ef114e792634d5684947866c4594c4e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/7ef114e792634d5684947866c4594c4e 2023-04-19 07:56:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:37,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:37,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:37,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:37,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:37,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:37,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:37,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:37,064 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:37,064 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:37,064 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:37,064 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:37,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:37,066 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,066 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:37,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:37,066 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,066 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,067 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/7ef114e792634d5684947866c4594c4e, entries=150, sequenceid=366, filesize=11.9 K 2023-04-19 07:56:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:37,068 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/5750e6e7ddd849d8ba1655503427b5c0 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5750e6e7ddd849d8ba1655503427b5c0 2023-04-19 07:56:37,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:37,069 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:37,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d87e539[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:37,069 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,070 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,070 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:37,070 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:37,070 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:37,070 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,070 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:37,071 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5750e6e7ddd849d8ba1655503427b5c0, entries=150, sequenceid=366, filesize=11.9 K 2023-04-19 07:56:37,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:37,072 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 051c522ed965d63912b88bb381b9d1d2 in 464ms, sequenceid=366, compaction requested=true 2023-04-19 07:56:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:37,072 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:37,073 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:37,073 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:37,073 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:37,073 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:56:37,073 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:56:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,074 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104696 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:56:37,074 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/A is initiating minor compaction (all files) 2023-04-19 07:56:37,074 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/A in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:37,074 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/98d84c6d1bd7466686f7a928e3b2e7e9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/3a2baedd34c743bcba794599f0d49e6c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/d91851fd4011453fb0ecbd6b39533abf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/6cdb3ee83c954680b0cd45886a6b0d6a] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=102.2 K 2023-04-19 07:56:37,075 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 98d84c6d1bd7466686f7a928e3b2e7e9, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681890994116 2023-04-19 07:56:37,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,075 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:56:37,075 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/B is initiating minor compaction (all files) 2023-04-19 07:56:37,075 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/B in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:37,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,075 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/db55488ce52f487397deb8c61da7b24e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/262ecd5001f34e07aa42f72fa646a464, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/e1458116ab42486fa82f77bf444ce7c5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/7ef114e792634d5684947866c4594c4e] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=48.1 K 2023-04-19 07:56:37,075 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 3a2baedd34c743bcba794599f0d49e6c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681890994221 2023-04-19 07:56:37,076 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting db55488ce52f487397deb8c61da7b24e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681890994116 2023-04-19 07:56:37,076 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting d91851fd4011453fb0ecbd6b39533abf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681890995410 2023-04-19 07:56:37,076 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 262ecd5001f34e07aa42f72fa646a464, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681890994221 2023-04-19 07:56:37,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:37,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:37,076 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:37,077 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:37,077 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:37,077 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:37,077 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 6cdb3ee83c954680b0cd45886a6b0d6a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681890996596 2023-04-19 07:56:37,077 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:37,077 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting e1458116ab42486fa82f77bf444ce7c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681890995410 2023-04-19 07:56:37,077 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7ef114e792634d5684947866c4594c4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681890996596 2023-04-19 07:56:37,080 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,080 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:37,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,080 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:37,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:37,081 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:37,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,083 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:37,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:37,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:37,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:37,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:37,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:37,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:37,086 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:37,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,086 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,086 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:37,087 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:37,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:37,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:37,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:37,089 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#B#compaction#555 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:37,093 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#A#compaction#554 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:37,187 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:37,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:37,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:37,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:37,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:37,190 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:37,190 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:37,190 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:37,190 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:37,191 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,191 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,192 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:37,192 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:37,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:37,193 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,193 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:37,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:37,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:37,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:37,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:37,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:37,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:37,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:37,194 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:37,194 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:37,194 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:37,194 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-19 07:56:37,195 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:37,195 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:37,195 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:37,195 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:37,195 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:37,195 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:37,213 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419e86a778957ab44c1a6d6b899590ceab1 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e86a778957ab44c1a6d6b899590ceab1 2023-04-19 07:56:37,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:37,348 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:37,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:37,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891057370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:37,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891057475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,509 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/bde49fe3bbff4810917d81d3badffcfa as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/bde49fe3bbff4810917d81d3badffcfa 2023-04-19 07:56:37,514 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/B of 051c522ed965d63912b88bb381b9d1d2 into bde49fe3bbff4810917d81d3badffcfa(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:37,515 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:37,515 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/B, priority=12, startTime=1681890997073; duration=0sec 2023-04-19 07:56:37,515 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:37,515 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-19 07:56:37,517 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-19 07:56:37,518 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/C is initiating minor compaction (all files) 2023-04-19 07:56:37,518 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/C in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:37,518 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/a1692cdd1e0e4fbcbe338066db255461, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/cf473e537bad420598b81c058c48d831, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/da445a01280d475fafd06b7d1ca4ff74, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5750e6e7ddd849d8ba1655503427b5c0] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=48.1 K 2023-04-19 07:56:37,518 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting a1692cdd1e0e4fbcbe338066db255461, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1681890994116 2023-04-19 07:56:37,518 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting cf473e537bad420598b81c058c48d831, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1681890994221 2023-04-19 07:56:37,520 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting da445a01280d475fafd06b7d1ca4ff74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681890995410 2023-04-19 07:56:37,520 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 5750e6e7ddd849d8ba1655503427b5c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681890996596 2023-04-19 07:56:37,528 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#C#compaction#557 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:37,536 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/9b13fdbabb8c457fbb7d4e27c2f58881 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/9b13fdbabb8c457fbb7d4e27c2f58881 2023-04-19 07:56:37,539 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/C of 051c522ed965d63912b88bb381b9d1d2 into 9b13fdbabb8c457fbb7d4e27c2f58881(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:37,539 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:37,539 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/C, priority=12, startTime=1681890997073; duration=0sec 2023-04-19 07:56:37,539 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:37,625 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=403, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/eb68b9cf996142ee81c04a009c224a0e 2023-04-19 07:56:37,635 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/2f8c587766744d87871905cf339db55b 2023-04-19 07:56:37,652 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/5684eb70047140e3b86ab598e6545608 2023-04-19 07:56:37,658 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/eb68b9cf996142ee81c04a009c224a0e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/eb68b9cf996142ee81c04a009c224a0e 2023-04-19 07:56:37,661 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/eb68b9cf996142ee81c04a009c224a0e, entries=150, sequenceid=403, filesize=25.4 K 2023-04-19 07:56:37,661 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/2f8c587766744d87871905cf339db55b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/2f8c587766744d87871905cf339db55b 2023-04-19 07:56:37,665 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/2f8c587766744d87871905cf339db55b, entries=150, sequenceid=403, filesize=11.9 K 2023-04-19 07:56:37,666 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/5684eb70047140e3b86ab598e6545608 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5684eb70047140e3b86ab598e6545608 2023-04-19 07:56:37,670 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5684eb70047140e3b86ab598e6545608, entries=150, sequenceid=403, filesize=11.9 K 2023-04-19 07:56:37,670 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 051c522ed965d63912b88bb381b9d1d2 in 476ms, sequenceid=403, compaction requested=false 2023-04-19 07:56:37,670 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:37,670 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:37,670 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:37,671 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:37,671 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,672 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:37,672 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:37,673 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,673 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:37,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:37,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:37,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:37,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:37,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:37,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e5f3392[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:37,675 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:37,675 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,677 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,677 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:37,678 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:37,678 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:37,678 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,678 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:37,679 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,679 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:37,679 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:37,679 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:37,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:37,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:37,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:37,682 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:37,682 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:37,682 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:37,682 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:37,682 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:37,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:37,683 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 07:56:37,683 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:37,683 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:37,683 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:37,683 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:37,683 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,684 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:37,684 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,684 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:37,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,684 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:37,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:37,685 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:37,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,687 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:37,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:37,690 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:37,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:37,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:37,690 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,690 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,690 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,690 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:37,690 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:37,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:37,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:37,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:37,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:37,695 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304191392853bab9a4798ba89ed06beec704a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191392853bab9a4798ba89ed06beec704a 2023-04-19 07:56:37,703 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=416, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/ddd06ab0484e4dc59ee67cd1b5b8f186 2023-04-19 07:56:37,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890997736, max timestamp=1681890997754Num uniques 859; 2023-04-19 07:56:37,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:37,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890997736, max timestamp=1681890997754Num uniques 859; 2023-04-19 07:56:37,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:37,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681890997736, max timestamp=1681890997754Num uniques 859; 2023-04-19 07:56:37,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:37,790 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:37,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:37,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:37,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:37,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:37,793 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:37,793 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:37,793 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:37,793 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:37,794 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,794 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,796 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:37,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:37,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:37,796 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,797 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,797 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:37,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:37,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:37,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:37,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:37,797 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:37,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:37,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:37,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:37,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:37,798 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:37,798 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:37,798 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:37,798 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:37,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:37,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:37,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:37,798 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:37,800 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:37,800 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:37,800 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,801 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:37,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:37,802 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:37,802 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@699a4814[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:37,810 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,810 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,810 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:37,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,811 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:37,811 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:37,811 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:37,813 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:37,813 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,813 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:37,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:37,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:37,813 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:37,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:37,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:37,816 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:37,816 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:37,816 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:37,816 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:37,816 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:37,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,818 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:37,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:37,818 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:37,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,820 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,822 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,822 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,822 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,822 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:37,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:37,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:37,822 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,822 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,822 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:37,822 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,822 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:37,822 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:37,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:37,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:37,822 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:37,822 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:37,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:37,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:37,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891057828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,922 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:37,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:37,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:37,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:37,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:37,926 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:37,926 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:37,926 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:37,926 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:37,926 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,926 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,928 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/cbdb5db2e4cf4c7581c531f5dae75716 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/cbdb5db2e4cf4c7581c531f5dae75716 2023-04-19 07:56:37,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:37,928 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:37,928 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,929 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:37,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:37,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:37,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:37,930 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:37,930 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:37,930 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:37,930 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:37,930 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:37,930 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:37,930 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:37,930 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:37,930 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:37,930 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:37,930 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:37,930 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,931 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:37,931 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:37,933 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,933 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:37,933 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/A of 051c522ed965d63912b88bb381b9d1d2 into cbdb5db2e4cf4c7581c531f5dae75716(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:37,933 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:37,933 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/A, priority=12, startTime=1681890997073; duration=0sec 2023-04-19 07:56:37,934 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:37,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,934 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:37,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:37,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891057932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:37,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:37,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@522a7c30[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:37,938 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:37,938 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,939 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,939 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:37,939 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:37,939 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:37,939 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,939 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:37,940 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:37,940 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:37,940 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,940 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:37,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:37,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:37,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:37,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:37,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:37,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:37,943 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:37,943 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:37,943 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:37,943 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:37,943 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:37,945 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,945 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,945 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:37,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,945 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:37,946 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,949 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,949 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,949 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:37,949 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:37,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:37,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:37,949 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:37,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:37,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:37,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:37,949 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:37,949 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:37,950 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:37,950 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:37,950 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:37,950 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:37,950 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:37,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:38,050 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:38,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:38,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:38,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:38,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:38,053 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:38,054 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:38,054 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:38,054 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:38,054 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,054 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,055 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,055 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:38,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:38,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:38,056 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,056 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:38,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:38,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:38,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:38,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:38,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:38,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:38,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:38,057 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:38,057 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:38,058 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:38,058 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:38,058 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:38,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:38,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:38,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:38,058 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:38,059 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,059 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:38,060 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,060 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:38,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:38,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:38,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:38,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:38,061 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:38,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20f6e561[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:38,061 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,062 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,062 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:38,062 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:38,062 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:38,062 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,062 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:38,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:38,064 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:38,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:38,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:38,067 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:38,067 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:38,067 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:38,067 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:38,067 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:38,068 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,068 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,068 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:38,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:38,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:38,069 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,070 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,072 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:38,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:38,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:38,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:38,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:38,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:38,073 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:38,073 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:38,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:38,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:38,073 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/ef52e2b245b64b20a8b32ca4012ac55f 2023-04-19 07:56:38,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:38,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891058137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,173 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:38,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:38,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:38,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:38,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:38,176 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:38,176 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:38,176 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:38,176 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:38,177 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,177 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,178 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,178 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:38,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,178 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,178 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:38,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:38,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:38,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:38,179 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:38,179 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:38,179 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:38,180 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:38,180 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:38,180 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:38,180 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:38,180 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:38,180 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:38,180 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:38,180 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:38,180 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,180 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:38,180 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:38,184 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,184 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:38,185 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,185 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:38,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:38,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37a2914e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:38,187 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:38,187 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,188 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:38,188 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:38,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:38,189 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:38,190 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:38,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:38,192 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:38,192 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:38,192 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:38,192 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:38,192 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:38,196 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,196 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:38,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,196 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:38,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:38,197 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,198 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:38,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,202 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,202 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:38,202 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:38,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:38,202 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:38,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:38,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:38,202 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:38,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:38,202 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:38,202 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,203 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,203 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,203 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,203 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:38,203 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:38,303 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:38,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:38,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:38,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:38,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:38,306 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:38,306 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:38,307 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:38,307 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:38,307 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,307 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,308 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:38,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:38,308 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,309 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,309 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:38,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:38,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:38,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:38,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:38,310 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:38,310 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:38,310 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:38,311 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:38,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:38,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:38,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:38,311 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:38,312 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,312 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:38,312 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,313 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:38,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:38,314 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:38,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cc3a546[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:38,314 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,315 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,316 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:38,316 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:38,316 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:38,316 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,316 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:38,318 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:38,319 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:38,319 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:38,319 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:38,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:38,321 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:38,321 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:38,321 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:38,321 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:38,321 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:38,323 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,323 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:38,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,323 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:38,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,324 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:38,326 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,328 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:38,328 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:38,328 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,328 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,328 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:38,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:38,328 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,328 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,328 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:38,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:38,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:38,328 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,329 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:38,329 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:38,329 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:38,429 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:38,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:38,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:38,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:38,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:38,432 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:38,432 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:38,432 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:38,432 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:38,433 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,433 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,434 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:38,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:38,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,434 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:38,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:38,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:38,435 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:38,435 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:38,435 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:38,436 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:38,436 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:38,436 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:38,436 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:38,436 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:38,436 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:38,436 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:38,436 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:38,436 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,436 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:38,436 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:38,441 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,441 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:38,442 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,442 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:38,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:38,443 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:38,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d898312[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:38,443 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:38,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891058441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,444 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,445 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:38,445 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:38,445 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:38,445 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,445 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:38,446 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,446 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:38,446 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:38,446 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:38,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:38,449 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:38,449 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:38,449 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:38,449 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:38,449 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:38,450 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,450 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,450 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:38,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:38,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:38,451 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:38,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:38,453 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:38,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:38,456 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,456 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,456 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,456 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:38,456 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:38,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:38,456 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:38,456 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,456 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:38,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:38,456 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:38,457 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,457 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:38,457 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:38,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:38,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:38,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:38,521 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/e6255424d1bf4de4969f3b55385f95c9 2023-04-19 07:56:38,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/ddd06ab0484e4dc59ee67cd1b5b8f186 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ddd06ab0484e4dc59ee67cd1b5b8f186 2023-04-19 07:56:38,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ddd06ab0484e4dc59ee67cd1b5b8f186, entries=200, sequenceid=416, filesize=32.3 K 2023-04-19 07:56:38,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/ef52e2b245b64b20a8b32ca4012ac55f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ef52e2b245b64b20a8b32ca4012ac55f 2023-04-19 07:56:38,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ef52e2b245b64b20a8b32ca4012ac55f, entries=150, sequenceid=416, filesize=11.9 K 2023-04-19 07:56:38,535 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/e6255424d1bf4de4969f3b55385f95c9 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e6255424d1bf4de4969f3b55385f95c9 2023-04-19 07:56:38,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e6255424d1bf4de4969f3b55385f95c9, entries=150, sequenceid=416, filesize=11.9 K 2023-04-19 07:56:38,542 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 051c522ed965d63912b88bb381b9d1d2 in 859ms, sequenceid=416, compaction requested=true 2023-04-19 07:56:38,542 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:38,544 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:38,544 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:38,544 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:38,544 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:38,544 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:38,549 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85826 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:38,549 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/A is initiating minor compaction (all files) 2023-04-19 07:56:38,549 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/A in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:38,549 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/cbdb5db2e4cf4c7581c531f5dae75716, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/eb68b9cf996142ee81c04a009c224a0e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ddd06ab0484e4dc59ee67cd1b5b8f186] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=83.8 K 2023-04-19 07:56:38,549 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:38,549 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting cbdb5db2e4cf4c7581c531f5dae75716, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681890996596 2023-04-19 07:56:38,549 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/B is initiating minor compaction (all files) 2023-04-19 07:56:38,550 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/B in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:38,550 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/bde49fe3bbff4810917d81d3badffcfa, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/2f8c587766744d87871905cf339db55b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ef52e2b245b64b20a8b32ca4012ac55f] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=36.3 K 2023-04-19 07:56:38,551 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting eb68b9cf996142ee81c04a009c224a0e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681890996724 2023-04-19 07:56:38,551 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting bde49fe3bbff4810917d81d3badffcfa, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681890996596 2023-04-19 07:56:38,551 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting ddd06ab0484e4dc59ee67cd1b5b8f186, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1681890997362 2023-04-19 07:56:38,551 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 2f8c587766744d87871905cf339db55b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681890996724 2023-04-19 07:56:38,552 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting ef52e2b245b64b20a8b32ca4012ac55f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1681890997362 2023-04-19 07:56:38,557 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:38,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:38,559 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#B#compaction#564 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:38,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:38,562 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#A#compaction#563 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:38,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:38,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:38,563 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:38,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:38,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:38,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:38,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,563 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,564 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,564 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:38,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:38,565 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:38,565 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:38,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:38,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:38,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:38,572 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:38,572 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:38,572 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:38,576 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:38,576 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:38,576 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:38,576 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:38,576 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-19 07:56:38,577 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:38,577 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:38,577 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:38,577 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:38,577 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:38,577 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:38,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:38,950 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:38,970 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/ef5dd9699cdc452d9aa08799ab18f99d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ef5dd9699cdc452d9aa08799ab18f99d 2023-04-19 07:56:38,974 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/B of 051c522ed965d63912b88bb381b9d1d2 into ef5dd9699cdc452d9aa08799ab18f99d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:38,974 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:38,974 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/B, priority=13, startTime=1681890998544; duration=0sec 2023-04-19 07:56:38,974 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:38,974 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:38,975 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:38,975 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/C is initiating minor compaction (all files) 2023-04-19 07:56:38,975 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/C in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:38,976 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/9b13fdbabb8c457fbb7d4e27c2f58881, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5684eb70047140e3b86ab598e6545608, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e6255424d1bf4de4969f3b55385f95c9] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=36.3 K 2023-04-19 07:56:38,976 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 9b13fdbabb8c457fbb7d4e27c2f58881, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681890996596 2023-04-19 07:56:38,977 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 5684eb70047140e3b86ab598e6545608, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681890996724 2023-04-19 07:56:38,977 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting e6255424d1bf4de4969f3b55385f95c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1681890997362 2023-04-19 07:56:38,989 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#C#compaction#566 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:38,994 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041919d79781b3c64e27bd9efc82d674d41c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041919d79781b3c64e27bd9efc82d674d41c 2023-04-19 07:56:39,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:39,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891058996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,019 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/0e466ac215544f939020430488ebb1e0 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/0e466ac215544f939020430488ebb1e0 2023-04-19 07:56:39,023 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/C of 051c522ed965d63912b88bb381b9d1d2 into 0e466ac215544f939020430488ebb1e0(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:39,024 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:39,024 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/C, priority=13, startTime=1681890998544; duration=0sec 2023-04-19 07:56:39,024 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:39,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:39,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891059115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:39,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891059317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,418 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=453, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/047e4174ea864945ae494fb11bffc1d5 2023-04-19 07:56:39,423 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/16ecb23236ac496b996f2868783d6c99 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/16ecb23236ac496b996f2868783d6c99 2023-04-19 07:56:39,428 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/A of 051c522ed965d63912b88bb381b9d1d2 into 16ecb23236ac496b996f2868783d6c99(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:39,428 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:39,428 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/A, priority=13, startTime=1681890998542; duration=0sec 2023-04-19 07:56:39,429 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:39,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:39,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891059620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,832 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/844cb4024b5445008a408a37dc738167 2023-04-19 07:56:39,842 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/5dc86d0b7b9c4e479f63cd1752d7a94e 2023-04-19 07:56:39,846 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/047e4174ea864945ae494fb11bffc1d5 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/047e4174ea864945ae494fb11bffc1d5 2023-04-19 07:56:39,850 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/047e4174ea864945ae494fb11bffc1d5, entries=150, sequenceid=453, filesize=25.4 K 2023-04-19 07:56:39,851 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/844cb4024b5445008a408a37dc738167 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/844cb4024b5445008a408a37dc738167 2023-04-19 07:56:39,855 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/844cb4024b5445008a408a37dc738167, entries=150, sequenceid=453, filesize=11.9 K 2023-04-19 07:56:39,864 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/5dc86d0b7b9c4e479f63cd1752d7a94e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5dc86d0b7b9c4e479f63cd1752d7a94e 2023-04-19 07:56:39,868 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5dc86d0b7b9c4e479f63cd1752d7a94e, entries=150, sequenceid=453, filesize=11.9 K 2023-04-19 07:56:39,869 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 051c522ed965d63912b88bb381b9d1d2 in 1293ms, sequenceid=453, compaction requested=false 2023-04-19 07:56:39,869 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:39,869 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:39,869 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:39,869 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:39,869 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,869 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:39,869 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:39,870 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:39,870 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:39,871 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:39,871 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:39,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:39,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:39,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a43b22f[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:39,872 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:39,872 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:39,874 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:39,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:39,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:39,874 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:39,874 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:39,874 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:39,874 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,874 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:39,876 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,876 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:39,876 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:39,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:39,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:39,876 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:39,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:39,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:39,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:39,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:39,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:39,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:39,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:39,879 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:39,879 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:39,879 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:39,879 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:39,879 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:39,881 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:39,881 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:39,881 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:39,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:39,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:39,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:39,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:39,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:39,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:39,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:39,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:39,881 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:39,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:39,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:39,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:39,883 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:39,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:39,888 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,888 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:39,888 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:39,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:39,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:39,888 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:39,888 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:39,888 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:39,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:39,888 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:39,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:39,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:39,888 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:39,888 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,888 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:39,889 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:39,889 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:39,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:39,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:39,988 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:39,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:39,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:39,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:39,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:39,992 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:39,992 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:39,992 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:39,992 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:39,993 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:39,993 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:39,996 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:39,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:39,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:39,996 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:39,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:39,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:39,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:39,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:39,997 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:39,997 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:39,998 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:39,998 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:39,998 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:39,998 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:39,998 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:39,998 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 07:56:39,998 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:39,998 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:39,998 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:39,998 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:39,999 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:39,999 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:40,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:40,128 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:40,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681891000158, max timestamp=1681891000179Num uniques 859; 2023-04-19 07:56:40,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:40,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681891000158, max timestamp=1681891000179Num uniques 859; 2023-04-19 07:56:40,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:40,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681891000158, max timestamp=1681891000179Num uniques 859; 2023-04-19 07:56:40,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-19 07:56:40,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:40,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891060231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:40,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891060334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,419 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304194b4e31fd28a24a19a9dda8deec19338f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194b4e31fd28a24a19a9dda8deec19338f 2023-04-19 07:56:40,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:40,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891060536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,824 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=467, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/06e0cb2a06d04970a52833f542d669bb 2023-04-19 07:56:40,834 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/e87deb90cc4f48e48685f78e02160e57 2023-04-19 07:56:40,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:40,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891060839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,844 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/e2ee022827d24504a01b09b899fb14f8 2023-04-19 07:56:40,847 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/06e0cb2a06d04970a52833f542d669bb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/06e0cb2a06d04970a52833f542d669bb 2023-04-19 07:56:40,850 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/06e0cb2a06d04970a52833f542d669bb, entries=150, sequenceid=467, filesize=25.4 K 2023-04-19 07:56:40,851 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/e87deb90cc4f48e48685f78e02160e57 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/e87deb90cc4f48e48685f78e02160e57 2023-04-19 07:56:40,855 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/e87deb90cc4f48e48685f78e02160e57, entries=150, sequenceid=467, filesize=11.9 K 2023-04-19 07:56:40,855 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/e2ee022827d24504a01b09b899fb14f8 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e2ee022827d24504a01b09b899fb14f8 2023-04-19 07:56:40,858 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e2ee022827d24504a01b09b899fb14f8, entries=150, sequenceid=467, filesize=11.9 K 2023-04-19 07:56:40,859 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.08 KB/130125 for 051c522ed965d63912b88bb381b9d1d2 in 861ms, sequenceid=467, compaction requested=true 2023-04-19 07:56:40,859 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:40,859 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:40,859 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:40,859 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:40,859 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:40,860 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:40,861 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,861 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:40,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:40,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:40,862 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:40,862 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:40,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:40,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@394878c8[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:40,863 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:40,863 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:40,864 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:40,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:40,865 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:40,865 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:40,865 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:40,865 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,866 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:40,866 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:40,866 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:40,867 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:40,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:40,869 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:40,869 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:40,869 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:40,869 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:40,869 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:40,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:40,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:40,872 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:40,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:40,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:40,873 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,874 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,876 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,876 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:40,876 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:40,876 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,876 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:40,876 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:40,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:40,877 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:40,876 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:40,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:40,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:40,877 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:40,877 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:40,877 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:40,877 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:40,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:40,977 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:40,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:40,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:40,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:40,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:40,980 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:40,980 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:40,981 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:40,981 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:40,981 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:40,981 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:40,982 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:40,982 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:40,983 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:40,983 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:40,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:40,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:40,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:40,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:40,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:40,984 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:40,984 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:40,984 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:40,984 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:40,984 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:40,984 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:40,984 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:40,984 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=127.08 KB heapSize=364.15 KB 2023-04-19 07:56:40,984 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:40,984 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:40,985 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:40,985 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:40,985 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:40,985 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:41,031 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-19 07:56:41,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:41,346 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:41,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:41,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891061376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:41,397 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304197275298f1e2641ae87359aa0b6188fc8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197275298f1e2641ae87359aa0b6188fc8 2023-04-19 07:56:41,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:41,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891061480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:41,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:41,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891061681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:41,803 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=503, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/8ac9d4036c864b9db901f67415437d26 2023-04-19 07:56:41,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:41,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891061986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,225 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/557eac5ae91d45029cd3c708e72752a4 2023-04-19 07:56:42,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:42,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891062492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,636 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/e264db1687814b5293c572d90daf3a5e 2023-04-19 07:56:42,639 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/8ac9d4036c864b9db901f67415437d26 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/8ac9d4036c864b9db901f67415437d26 2023-04-19 07:56:42,643 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/8ac9d4036c864b9db901f67415437d26, entries=150, sequenceid=503, filesize=25.4 K 2023-04-19 07:56:42,643 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/557eac5ae91d45029cd3c708e72752a4 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/557eac5ae91d45029cd3c708e72752a4 2023-04-19 07:56:42,646 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/557eac5ae91d45029cd3c708e72752a4, entries=150, sequenceid=503, filesize=11.9 K 2023-04-19 07:56:42,647 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/e264db1687814b5293c572d90daf3a5e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e264db1687814b5293c572d90daf3a5e 2023-04-19 07:56:42,650 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e264db1687814b5293c572d90daf3a5e, entries=150, sequenceid=503, filesize=11.9 K 2023-04-19 07:56:42,651 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.08 KB/130125, heapSize ~364.10 KB/372840, currentSize=53.67 KB/54960 for 051c522ed965d63912b88bb381b9d1d2 in 1667ms, sequenceid=503, compaction requested=true 2023-04-19 07:56:42,651 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:42,651 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:42,651 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:42,651 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:42,651 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,651 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:42,651 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:42,653 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:42,653 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:42,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:42,653 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:42,653 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:42,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:42,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:42,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:42,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:42,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:42,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28300332[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:42,655 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:42,655 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:42,656 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:42,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:42,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:42,656 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:42,656 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:42,656 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:42,657 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,657 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:42,658 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:42,658 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:42,658 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,658 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:42,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:42,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:42,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:42,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:42,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:42,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:42,660 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:42,660 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:42,660 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:42,660 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:42,660 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:42,663 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:42,663 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:42,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:42,663 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:42,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:42,664 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:42,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:42,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,665 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:42,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,668 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,668 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:42,668 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:42,668 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:42,668 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:42,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:42,668 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:42,668 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:42,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:42,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:42,668 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:42,668 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:42,668 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:42,668 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:42,669 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:42,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:42,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:42,769 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:42,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:42,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:42,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:42,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:42,772 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:42,772 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:42,773 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:42,773 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:42,773 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:42,773 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:42,774 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,774 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:42,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:42,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:42,775 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:42,775 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:42,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:42,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:42,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:42,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:42,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:42,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:42,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:42,776 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:42,776 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:42,776 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:42,776 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:42,776 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:42,776 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:42,776 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:42,776 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-19 07:56:42,776 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:42,777 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:42,777 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:42,777 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:42,777 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:42,777 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:43,198 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041941de455fcfec4ee8a00b6540fe933322 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041941de455fcfec4ee8a00b6540fe933322 2023-04-19 07:56:43,204 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=514, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/ad51e73c10574cd6ad40efabf3954cbf 2023-04-19 07:56:43,215 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/27309727138f4b8695017b354c082ebb 2023-04-19 07:56:43,225 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/492ec3de141246feb1c4a146815d5837 2023-04-19 07:56:43,230 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/ad51e73c10574cd6ad40efabf3954cbf as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ad51e73c10574cd6ad40efabf3954cbf 2023-04-19 07:56:43,243 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ad51e73c10574cd6ad40efabf3954cbf, entries=150, sequenceid=514, filesize=25.4 K 2023-04-19 07:56:43,244 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/27309727138f4b8695017b354c082ebb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/27309727138f4b8695017b354c082ebb 2023-04-19 07:56:43,248 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/27309727138f4b8695017b354c082ebb, entries=150, sequenceid=514, filesize=11.9 K 2023-04-19 07:56:43,248 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/492ec3de141246feb1c4a146815d5837 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/492ec3de141246feb1c4a146815d5837 2023-04-19 07:56:43,252 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/492ec3de141246feb1c4a146815d5837, entries=150, sequenceid=514, filesize=11.9 K 2023-04-19 07:56:43,253 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 051c522ed965d63912b88bb381b9d1d2 in 477ms, sequenceid=514, compaction requested=true 2023-04-19 07:56:43,253 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:43,253 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:43,255 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,255 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,255 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,255 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:43,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69e09435[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:43,257 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:43,257 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,260 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,260 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:43,260 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:43,260 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:43,261 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:43,261 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:43,262 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:43,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,272 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:43,273 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:43,273 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:43,273 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:43,273 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:43,273 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:43,273 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:43,274 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,274 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,274 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:43,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:43,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:43,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,276 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,277 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,283 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,283 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:43,283 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,283 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:43,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:43,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:43,283 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:43,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:43,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:43,283 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:43,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:43,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,284 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,284 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:43,284 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:43,384 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:43,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:43,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:43,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:43,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:43,389 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:43,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:43,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:43,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:43,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,389 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,390 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:43,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:43,391 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,391 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,391 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:43,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:43,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:43,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:43,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:43,392 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:43,392 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:43,392 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:43,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:43,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:43,393 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:43,393 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:43,393 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:43,393 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:43,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:43,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:43,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:43,393 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:43,395 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,395 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,395 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,395 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:43,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e3591ec[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:43,397 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:43,397 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,398 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,398 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:43,399 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:43,399 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:43,399 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,399 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:43,401 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,401 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:43,401 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,401 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:43,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:43,404 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:43,404 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:43,404 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:43,404 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:43,404 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:43,405 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,405 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,406 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:43,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:43,406 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,407 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,410 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,410 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:43,410 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:43,410 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,410 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,410 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,410 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,410 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,410 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:43,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:43,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:43,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:43,410 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:43,411 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:43,411 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:43,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:43,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:43,511 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:43,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:43,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:43,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:43,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:43,515 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:43,515 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:43,515 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:43,515 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:43,515 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,515 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,516 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,517 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:43,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:43,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:43,517 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,517 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:43,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:43,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:43,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:43,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:43,519 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:43,519 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:43,519 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:43,519 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:43,519 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:43,519 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:43,520 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:43,520 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:43,520 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:43,520 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:43,520 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:43,520 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,520 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:43,520 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:43,523 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,523 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,523 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,523 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:43,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c13ada[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:43,525 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:43,525 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,526 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,526 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:43,526 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:43,526 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:43,526 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,526 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:43,528 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,528 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:43,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:43,531 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:43,531 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:43,531 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:43,531 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:43,531 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:43,533 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,534 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,536 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:43,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:43,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:43,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,537 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:43,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:43,541 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:43,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:43,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:43,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:43,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:43,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,541 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:43,541 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:43,541 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,542 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,542 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,542 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:43,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:43,542 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,542 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:43,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:43,578 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-19 07:56:43,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:43,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:43,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:43,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:43,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:43,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:43,597 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041981e5cb3eebf6419ca6907287ee6f73bf to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041981e5cb3eebf6419ca6907287ee6f73bf 2023-04-19 07:56:43,609 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=526, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/79be28dd10ff45b68fa76f195cabee2b 2023-04-19 07:56:43,629 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/4e4ba1e6f23c4085a42c169f38ad02d7 2023-04-19 07:56:43,642 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:43,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:43,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:43,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:43,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:43,656 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=526 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/2270fdd017cf4df6a605d287f05320ea 2023-04-19 07:56:43,656 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:43,656 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:43,656 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:43,656 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:43,657 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,657 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,660 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:43,660 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:43,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:43,661 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,661 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:43,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681891003630, max timestamp=1681891003659Num uniques 859; 2023-04-19 07:56:43,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:43,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:43,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:43,663 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/79be28dd10ff45b68fa76f195cabee2b as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/79be28dd10ff45b68fa76f195cabee2b 2023-04-19 07:56:43,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681891003630, max timestamp=1681891003659Num uniques 859; 2023-04-19 07:56:43,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:43,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:43,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:43,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681891003630, max timestamp=1681891003659Num uniques 859; 2023-04-19 07:56:43,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:43,665 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:43,665 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:43,665 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:43,666 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:43,666 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:43,666 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:43,666 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:43,666 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:43,666 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:43,666 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:43,667 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:43,667 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,667 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:43,667 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:43,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/79be28dd10ff45b68fa76f195cabee2b, entries=200, sequenceid=526, filesize=32.3 K 2023-04-19 07:56:43,668 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,668 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/4e4ba1e6f23c4085a42c169f38ad02d7 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4e4ba1e6f23c4085a42c169f38ad02d7 2023-04-19 07:56:43,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,669 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:43,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cc092e3[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:43,671 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:43,671 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,672 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,672 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:43,672 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,672 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:43,672 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:43,672 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:43,673 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4e4ba1e6f23c4085a42c169f38ad02d7, entries=150, sequenceid=526, filesize=11.9 K 2023-04-19 07:56:43,673 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/2270fdd017cf4df6a605d287f05320ea as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/2270fdd017cf4df6a605d287f05320ea 2023-04-19 07:56:43,675 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,675 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:43,675 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:43,675 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,676 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/2270fdd017cf4df6a605d287f05320ea, entries=150, sequenceid=526, filesize=11.9 K 2023-04-19 07:56:43,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,677 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=40.25 KB/41220 for 051c522ed965d63912b88bb381b9d1d2 in 99ms, sequenceid=526, compaction requested=true 2023-04-19 07:56:43,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,677 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:43,678 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:43,678 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 07:56:43,678 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 07:56:43,678 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:43,678 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:43,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:43,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:43,678 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:43,678 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:43,678 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:43,678 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:43,678 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:43,679 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73866 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 07:56:43,679 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/B is initiating minor compaction (all files) 2023-04-19 07:56:43,680 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/B in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:43,680 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ef5dd9699cdc452d9aa08799ab18f99d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/844cb4024b5445008a408a37dc738167, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/e87deb90cc4f48e48685f78e02160e57, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/557eac5ae91d45029cd3c708e72752a4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/27309727138f4b8695017b354c082ebb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4e4ba1e6f23c4085a42c169f38ad02d7] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=72.1 K 2023-04-19 07:56:43,680 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 164066 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 07:56:43,680 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/A is initiating minor compaction (all files) 2023-04-19 07:56:43,680 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/A in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:43,680 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/16ecb23236ac496b996f2868783d6c99, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/047e4174ea864945ae494fb11bffc1d5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/06e0cb2a06d04970a52833f542d669bb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/8ac9d4036c864b9db901f67415437d26, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ad51e73c10574cd6ad40efabf3954cbf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/79be28dd10ff45b68fa76f195cabee2b] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=160.2 K 2023-04-19 07:56:43,680 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting ef5dd9699cdc452d9aa08799ab18f99d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1681890997362 2023-04-19 07:56:43,681 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 16ecb23236ac496b996f2868783d6c99, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1681890997362 2023-04-19 07:56:43,681 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 844cb4024b5445008a408a37dc738167, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681890997783 2023-04-19 07:56:43,681 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 047e4174ea864945ae494fb11bffc1d5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681890997783 2023-04-19 07:56:43,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:43,681 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting e87deb90cc4f48e48685f78e02160e57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1681890998969 2023-04-19 07:56:43,681 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-19 07:56:43,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:43,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:43,682 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,682 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 557eac5ae91d45029cd3c708e72752a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1681891000220 2023-04-19 07:56:43,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,682 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:43,682 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 06e0cb2a06d04970a52833f542d669bb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1681890998969 2023-04-19 07:56:43,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:43,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:43,682 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:43,682 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 27309727138f4b8695017b354c082ebb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1681891001357 2023-04-19 07:56:43,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:43,682 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 8ac9d4036c864b9db901f67415437d26, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1681891000220 2023-04-19 07:56:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:43,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:43,683 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 4e4ba1e6f23c4085a42c169f38ad02d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1681891003567 2023-04-19 07:56:43,683 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,683 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting ad51e73c10574cd6ad40efabf3954cbf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1681891001357 2023-04-19 07:56:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,685 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 79be28dd10ff45b68fa76f195cabee2b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1681891003567 2023-04-19 07:56:43,685 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,688 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:43,688 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:43,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:43,688 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,688 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:43,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:43,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:43,688 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,689 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,689 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:43,689 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:43,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:43,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:43,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:43,698 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#B#compaction#583 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:43,698 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419f4a05e6b480b4dbeb726a7b3b3d93f0e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f4a05e6b480b4dbeb726a7b3b3d93f0e 2023-04-19 07:56:43,703 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#A#compaction#582 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:43,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681891003748, max timestamp=1681891003782Num uniques 859; 2023-04-19 07:56:43,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681891003748, max timestamp=1681891003782Num uniques 859; 2023-04-19 07:56:43,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:43,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681891003748, max timestamp=1681891003782Num uniques 859; 2023-04-19 07:56:43,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:43,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:43,789 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:43,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:43,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:43,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:43,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:43,793 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:43,793 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:43,793 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:43,793 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:43,793 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,793 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,794 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,794 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:43,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:43,795 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,795 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:43,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:43,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:43,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:43,796 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:43,796 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:43,796 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:43,796 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:43,796 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:43,796 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:43,796 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:43,796 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:43,796 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:43,796 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:43,796 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:43,796 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,796 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:43,797 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:43,801 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,801 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,802 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,802 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:43,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:43,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@516cf872[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:43,803 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:43,803 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,804 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,805 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:43,805 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:43,805 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:43,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,805 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:43,806 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,806 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:43,806 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,806 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:43,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:43,808 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:43,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:43,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:43,809 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:43,809 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:43,810 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,810 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,810 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:43,811 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,812 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,815 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,815 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,815 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:43,815 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:43,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:43,815 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:43,815 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:43,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:43,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:43,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:43,815 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:43,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:43,815 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,815 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,815 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:43,816 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:43,816 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:43,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 469 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891063866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,916 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:43,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:43,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:43,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:43,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:43,919 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:43,919 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:43,919 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:43,920 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:43,920 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,920 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,921 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:43,921 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:43,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:43,921 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,922 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:43,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:43,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:43,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:43,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:43,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:43,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:43,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:43,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:43,923 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:43,923 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:43,923 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:43,923 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:43,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:43,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:43,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:43,923 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:43,925 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,925 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,925 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,925 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:43,927 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:43,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1394b942[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:43,927 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,928 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,929 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:43,929 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:43,929 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:43,929 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,929 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:43,931 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,931 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,931 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:43,931 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:43,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:43,933 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:43,933 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:43,933 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:43,933 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:43,933 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:43,935 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,935 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,935 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:43,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:43,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:43,936 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:43,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,937 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:43,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:43,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,939 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:43,939 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:43,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:43,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:43,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:43,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:43,939 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,939 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:43,940 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,940 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:43,940 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:43,940 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:43,940 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:43,940 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,940 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:43,940 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:43,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:43,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:43,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:43,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:43,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 471 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891063971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,040 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:44,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:44,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:44,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:44,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:44,044 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:44,044 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:44,044 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:44,044 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:44,044 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,061 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,063 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:44,063 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:44,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:44,063 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,063 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:44,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:44,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:44,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:44,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:44,064 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:44,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:44,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:44,065 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:44,065 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:44,065 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:44,065 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:44,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:44,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:44,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:44,065 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:44,067 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,067 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,068 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:44,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e1e4600[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:44,069 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:44,070 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,071 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,071 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:44,071 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:44,071 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:44,071 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,071 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:44,072 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,072 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:44,072 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:44,072 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:44,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:44,074 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:44,074 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:44,074 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:44,074 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:44,075 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:44,076 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,076 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,076 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:44,077 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,078 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,081 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:44,081 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:44,081 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:44,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:44,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:44,081 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:44,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:44,082 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,082 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,082 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,082 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,082 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,082 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,082 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:44,082 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:44,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:44,113 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/5f8d0d0c02ec433b864f82ccb237f3bb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/5f8d0d0c02ec433b864f82ccb237f3bb 2023-04-19 07:56:44,116 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/B of 051c522ed965d63912b88bb381b9d1d2 into 5f8d0d0c02ec433b864f82ccb237f3bb(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:44,117 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:44,117 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/B, priority=10, startTime=1681891003678; duration=0sec 2023-04-19 07:56:44,117 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:44,117 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-19 07:56:44,118 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=551, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/461baeca889e45b5b7ca4244dbe70873 2023-04-19 07:56:44,119 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73832 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-19 07:56:44,119 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/C is initiating minor compaction (all files) 2023-04-19 07:56:44,119 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/C in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:44,119 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/0e466ac215544f939020430488ebb1e0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5dc86d0b7b9c4e479f63cd1752d7a94e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e2ee022827d24504a01b09b899fb14f8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e264db1687814b5293c572d90daf3a5e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/492ec3de141246feb1c4a146815d5837, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/2270fdd017cf4df6a605d287f05320ea] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=72.1 K 2023-04-19 07:56:44,119 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 0e466ac215544f939020430488ebb1e0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1681890997362 2023-04-19 07:56:44,120 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 5dc86d0b7b9c4e479f63cd1752d7a94e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681890997783 2023-04-19 07:56:44,120 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting e2ee022827d24504a01b09b899fb14f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1681890998969 2023-04-19 07:56:44,120 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting e264db1687814b5293c572d90daf3a5e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1681891000220 2023-04-19 07:56:44,121 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 492ec3de141246feb1c4a146815d5837, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1681891001357 2023-04-19 07:56:44,121 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 2270fdd017cf4df6a605d287f05320ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1681891003567 2023-04-19 07:56:44,140 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/37b712ce85224f90bf059bf0a31b7ab9 2023-04-19 07:56:44,147 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#C#compaction#585 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:44,152 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/c6f106f950824707b34f2245cafcaafd as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/c6f106f950824707b34f2245cafcaafd 2023-04-19 07:56:44,156 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/A of 051c522ed965d63912b88bb381b9d1d2 into c6f106f950824707b34f2245cafcaafd(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:44,156 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:44,156 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/A, priority=10, startTime=1681891003678; duration=0sec 2023-04-19 07:56:44,156 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:44,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/cb809e82e1394ca58be21d39411a6b29 2023-04-19 07:56:44,164 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/461baeca889e45b5b7ca4244dbe70873 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/461baeca889e45b5b7ca4244dbe70873 2023-04-19 07:56:44,167 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/461baeca889e45b5b7ca4244dbe70873, entries=200, sequenceid=551, filesize=32.3 K 2023-04-19 07:56:44,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/37b712ce85224f90bf059bf0a31b7ab9 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/37b712ce85224f90bf059bf0a31b7ab9 2023-04-19 07:56:44,170 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/7262aa52217e4a9ca1982d4f849ccf15 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/7262aa52217e4a9ca1982d4f849ccf15 2023-04-19 07:56:44,171 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/37b712ce85224f90bf059bf0a31b7ab9, entries=150, sequenceid=551, filesize=11.9 K 2023-04-19 07:56:44,172 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/cb809e82e1394ca58be21d39411a6b29 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/cb809e82e1394ca58be21d39411a6b29 2023-04-19 07:56:44,175 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/C of 051c522ed965d63912b88bb381b9d1d2 into 7262aa52217e4a9ca1982d4f849ccf15(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:44,175 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:44,175 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/C, priority=10, startTime=1681891003678; duration=0sec 2023-04-19 07:56:44,175 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:44,175 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/cb809e82e1394ca58be21d39411a6b29, entries=150, sequenceid=551, filesize=11.9 K 2023-04-19 07:56:44,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=134.18 KB/137400 for 051c522ed965d63912b88bb381b9d1d2 in 496ms, sequenceid=551, compaction requested=false 2023-04-19 07:56:44,177 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:44,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:44,179 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-19 07:56:44,179 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:44,179 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 07:56:44,179 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:44,179 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 07:56:44,179 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:44,179 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-19 07:56:44,182 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:44,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:44,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:44,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:44,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:44,187 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:44,187 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:44,187 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:44,187 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:44,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,188 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,189 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,189 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:44,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:44,189 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,189 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:44,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:44,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:44,193 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:44,193 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:44,193 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:44,196 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:44,196 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:44,196 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:44,196 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:44,198 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,198 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:44,200 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:44,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f18aaa[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:44,200 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,205 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,205 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,205 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:44,205 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:44,205 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:44,205 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:44,205 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,206 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:44,206 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419f73e43d74f8745ebb90a7cbd328d277c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f73e43d74f8745ebb90a7cbd328d277c 2023-04-19 07:56:44,206 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,207 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:44,207 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:44,207 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:44,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 480 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891064205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:44,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:44,211 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:44,211 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:44,211 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:44,211 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:44,211 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:44,213 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=592, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/f3e49958f97e401c906df7b4e4a42d89 2023-04-19 07:56:44,213 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,213 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:44,213 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,215 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,217 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,219 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:44,219 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,219 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,219 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:44,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:44,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:44,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:44,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:44,219 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:44,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:44,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:44,219 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,219 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,219 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,219 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,220 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,220 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,219 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:44,220 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:44,223 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=592 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/9485887385174e65812ba4db89c73853 2023-04-19 07:56:44,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:44,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] ipc.CallRunner(144): callId: 482 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891064309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,320 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:44,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:44,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:44,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:44,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:44,323 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:44,323 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:44,323 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:44,323 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:44,324 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,324 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,326 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,326 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:44,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:44,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:44,326 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,326 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:44,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:44,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:44,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:44,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:44,328 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:44,328 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:44,328 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:44,328 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:44,330 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,330 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,330 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,330 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:44,333 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:44,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@684b6e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:44,333 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,334 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,334 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:44,334 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:44,334 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:44,334 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,334 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:44,336 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:44,336 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:44,336 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,336 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:44,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:44,339 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:44,339 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:44,339 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:44,339 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:44,339 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:44,342 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,342 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,342 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:44,343 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,344 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,347 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,347 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:44,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:44,347 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,347 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:44,347 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:44,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:44,347 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:44,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,348 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,348 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:44,348 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:44,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:44,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:44,448 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:44,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:44,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:44,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:44,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:44,452 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:44,452 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:44,452 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:44,452 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:44,452 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,452 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,453 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,454 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:44,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:44,454 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,454 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:44,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:44,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:44,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:44,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:44,455 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:44,455 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:44,455 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:44,455 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:44,455 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:44,456 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:44,456 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:44,456 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:44,456 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:44,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:44,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:44,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:44,456 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:44,457 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,457 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,458 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,458 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:44,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3680bc88[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:44,459 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:44,460 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,462 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,462 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:44,462 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:44,463 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:44,463 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,463 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:44,464 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,464 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:44,464 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:44,464 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:44,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:44,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:44,467 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:44,467 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:44,467 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:44,467 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:44,467 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:44,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,469 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:44,470 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,471 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,473 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:44,473 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,474 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,474 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:44,474 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:44,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:44,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:44,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:44,474 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,474 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,474 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,474 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,474 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:44,474 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:44,474 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:44,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:44,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-19 07:56:44,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42359] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37938 deadline: 1681891064512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,574 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:44,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:44,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:44,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:44,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:44,578 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:44,578 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:44,578 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:44,578 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:44,578 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,578 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,581 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:44,581 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:44,581 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,581 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:44,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:44,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:44,582 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:44,582 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:44,582 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:44,582 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:44,582 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:44,583 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:44,583 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:44,583 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:44,583 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:44,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:44,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:44,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:44,583 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:44,584 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,584 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,584 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,585 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:44,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:44,586 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:44,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ad2c21e[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:44,586 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,587 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,587 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:44,587 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:44,587 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:44,587 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,587 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:44,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:44,592 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,592 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:44,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:44,595 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:44,595 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:44,595 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:44,595 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:44,595 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:44,597 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,597 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:44,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,598 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:44,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:44,598 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:44,599 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:44,601 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,601 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,601 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:44,601 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:44,601 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,601 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:44,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:44,602 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,602 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:44,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:44,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:44,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:44,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:44,602 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:44,602 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:44,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=592 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/e61bbedf3a964519912780dd4d79b4bb 2023-04-19 07:56:44,637 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/f3e49958f97e401c906df7b4e4a42d89 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/f3e49958f97e401c906df7b4e4a42d89 2023-04-19 07:56:44,640 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/f3e49958f97e401c906df7b4e4a42d89, entries=150, sequenceid=592, filesize=25.4 K 2023-04-19 07:56:44,641 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/9485887385174e65812ba4db89c73853 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9485887385174e65812ba4db89c73853 2023-04-19 07:56:44,644 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9485887385174e65812ba4db89c73853, entries=150, sequenceid=592, filesize=11.9 K 2023-04-19 07:56:44,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/e61bbedf3a964519912780dd4d79b4bb as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e61bbedf3a964519912780dd4d79b4bb 2023-04-19 07:56:44,648 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e61bbedf3a964519912780dd4d79b4bb, entries=150, sequenceid=592, filesize=11.9 K 2023-04-19 07:56:44,648 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 051c522ed965d63912b88bb381b9d1d2 in 469ms, sequenceid=592, compaction requested=true 2023-04-19 07:56:44,648 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:44,649 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:44,649 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:44,649 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:44,649 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-19 07:56:44,649 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:44,650 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86132 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:44,650 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:44,650 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/A is initiating minor compaction (all files) 2023-04-19 07:56:44,650 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/B is initiating minor compaction (all files) 2023-04-19 07:56:44,650 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/A in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:44,650 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/B in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:44,650 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/c6f106f950824707b34f2245cafcaafd, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/461baeca889e45b5b7ca4244dbe70873, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/f3e49958f97e401c906df7b4e4a42d89] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=84.1 K 2023-04-19 07:56:44,650 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/5f8d0d0c02ec433b864f82ccb237f3bb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/37b712ce85224f90bf059bf0a31b7ab9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9485887385174e65812ba4db89c73853] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=36.6 K 2023-04-19 07:56:44,650 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting c6f106f950824707b34f2245cafcaafd, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1681891003567 2023-04-19 07:56:44,651 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 5f8d0d0c02ec433b864f82ccb237f3bb, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1681891003567 2023-04-19 07:56:44,651 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting 461baeca889e45b5b7ca4244dbe70873, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681891003630 2023-04-19 07:56:44,651 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 37b712ce85224f90bf059bf0a31b7ab9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681891003671 2023-04-19 07:56:44,651 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] compactions.Compactor(207): Compacting f3e49958f97e401c906df7b4e4a42d89, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=592, earliestPutTs=1681891003854 2023-04-19 07:56:44,651 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 9485887385174e65812ba4db89c73853, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=592, earliestPutTs=1681891003854 2023-04-19 07:56:44,658 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#B#compaction#591 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:44,669 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#A#compaction#590 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:44,702 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:44,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:44,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-19 07:56:44,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-19 07:56:44,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-19 07:56:44,706 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-19 07:56:44,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:44,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-19 07:56:44,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-19 07:56:44,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,706 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,708 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:44,708 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:44,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:44,709 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:44,709 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-19 07:56:44,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:44,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-19 07:56:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:44,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-19 07:56:44,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-19 07:56:44,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-19 07:56:44,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-19 07:56:44,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-19 07:56:44,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-19 07:56:44,710 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-19 07:56:44,711 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-19 07:56:44,711 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:44,711 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. started... 2023-04-19 07:56:44,711 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-19 07:56:44,711 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:44,711 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:44,711 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:44,711 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:44,711 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:44,711 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:44,725 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304192a61bfb5a6a844d98ba8cc1ca3813da7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192a61bfb5a6a844d98ba8cc1ca3813da7 2023-04-19 07:56:44,737 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=601, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/d62e577018fc465780e7a7d088d6ee39 2023-04-19 07:56:44,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:44,820 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. as already flushing 2023-04-19 07:56:44,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681891004876, max timestamp=1681891004898Num uniques 859; 2023-04-19 07:56:44,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681891004876, max timestamp=1681891004898Num uniques 859; 2023-04-19 07:56:44,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681891004876, max timestamp=1681891004898Num uniques 859; 2023-04-19 07:56:44,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:44,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:44,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-19 07:56:44,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681891004941, max timestamp=1681891004970Num uniques 1018; 2023-04-19 07:56:44,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:56:44,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681891004941, max timestamp=1681891004970Num uniques 1018; 2023-04-19 07:56:44,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:56:44,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681891004941, max timestamp=1681891004970Num uniques 1018; 2023-04-19 07:56:44,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-19 07:56:45,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681891005018, max timestamp=1681891005037Num uniques 1027; 2023-04-19 07:56:45,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681891005018, max timestamp=1681891005037Num uniques 1027; 2023-04-19 07:56:45,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-19 07:56:45,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681891005018, max timestamp=1681891005037Num uniques 1027; 2023-04-19 07:56:45,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-19 07:56:45,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-19 07:56:45,060 DEBUG [Thread-3725] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23e65a76 to 127.0.0.1:58087 2023-04-19 07:56:45,060 DEBUG [Thread-3725] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:45,060 DEBUG [Thread-3721] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39241fe3 to 127.0.0.1:58087 2023-04-19 07:56:45,061 DEBUG [Thread-3721] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:45,064 DEBUG [Thread-3727] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e6a6f3c to 127.0.0.1:58087 2023-04-19 07:56:45,064 DEBUG [Thread-3727] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:45,065 DEBUG [Thread-3710] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3bafb41f to 127.0.0.1:58087 2023-04-19 07:56:45,065 DEBUG [Thread-3710] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:45,066 DEBUG [Thread-3719] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d759aab to 127.0.0.1:58087 2023-04-19 07:56:45,066 DEBUG [Thread-3719] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:45,066 DEBUG [Thread-3723] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbb5b46 to 127.0.0.1:58087 2023-04-19 07:56:45,066 DEBUG [Thread-3723] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:45,073 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/ba5597d18edb456fa754e612ea2a063a as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ba5597d18edb456fa754e612ea2a063a 2023-04-19 07:56:45,077 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/B of 051c522ed965d63912b88bb381b9d1d2 into ba5597d18edb456fa754e612ea2a063a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:45,077 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:45,077 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/B, priority=13, startTime=1681891004649; duration=0sec 2023-04-19 07:56:45,077 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:45,077 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-19 07:56:45,077 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-19 07:56:45,078 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1912): 051c522ed965d63912b88bb381b9d1d2/C is initiating minor compaction (all files) 2023-04-19 07:56:45,078 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 051c522ed965d63912b88bb381b9d1d2/C in TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:45,078 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/7262aa52217e4a9ca1982d4f849ccf15, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/cb809e82e1394ca58be21d39411a6b29, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e61bbedf3a964519912780dd4d79b4bb] into tmpdir=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp, totalSize=36.6 K 2023-04-19 07:56:45,078 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting 7262aa52217e4a9ca1982d4f849ccf15, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=526, earliestPutTs=1681891003567 2023-04-19 07:56:45,078 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting cb809e82e1394ca58be21d39411a6b29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1681891003671 2023-04-19 07:56:45,078 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] compactions.Compactor(207): Compacting e61bbedf3a964519912780dd4d79b4bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=592, earliestPutTs=1681891003854 2023-04-19 07:56:45,082 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] throttle.PressureAwareThroughputController(145): 051c522ed965d63912b88bb381b9d1d2#C#compaction#594 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-19 07:56:45,148 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/2f3439ed71f241d69d0e2b9833729419 2023-04-19 07:56:45,490 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/6988419e14854218ba908dd0a7df8114 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/6988419e14854218ba908dd0a7df8114 2023-04-19 07:56:45,492 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/da70f5aec25b4702865ac7daf1c3ec66 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/da70f5aec25b4702865ac7daf1c3ec66 2023-04-19 07:56:45,493 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/C of 051c522ed965d63912b88bb381b9d1d2 into 6988419e14854218ba908dd0a7df8114(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:45,493 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:45,493 INFO [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/C, priority=13, startTime=1681891004649; duration=0sec 2023-04-19 07:56:45,494 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:45,495 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 051c522ed965d63912b88bb381b9d1d2/A of 051c522ed965d63912b88bb381b9d1d2 into da70f5aec25b4702865ac7daf1c3ec66(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-19 07:56:45,496 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:45,496 INFO [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2., storeName=051c522ed965d63912b88bb381b9d1d2/A, priority=13, startTime=1681891004649; duration=0sec 2023-04-19 07:56:45,496 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:45,556 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/e78c756d805b4f0ab77d49b1316f1044 2023-04-19 07:56:45,560 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/d62e577018fc465780e7a7d088d6ee39 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/d62e577018fc465780e7a7d088d6ee39 2023-04-19 07:56:45,563 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/d62e577018fc465780e7a7d088d6ee39, entries=100, sequenceid=601, filesize=18.6 K 2023-04-19 07:56:45,563 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/2f3439ed71f241d69d0e2b9833729419 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/2f3439ed71f241d69d0e2b9833729419 2023-04-19 07:56:45,566 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/2f3439ed71f241d69d0e2b9833729419, entries=100, sequenceid=601, filesize=9.5 K 2023-04-19 07:56:45,567 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/e78c756d805b4f0ab77d49b1316f1044 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e78c756d805b4f0ab77d49b1316f1044 2023-04-19 07:56:45,571 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e78c756d805b4f0ab77d49b1316f1044, entries=100, sequenceid=601, filesize=9.5 K 2023-04-19 07:56:45,572 INFO [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=80.51 KB/82440 for 051c522ed965d63912b88bb381b9d1d2 in 861ms, sequenceid=601, compaction requested=false 2023-04-19 07:56:45,572 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:45,572 DEBUG [rs(jenkins-hbase4.apache.org,42359,1681890819191)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:45,572 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-19 07:56:45,572 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-19 07:56:45,572 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,572 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-19 07:56:45,572 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-19 07:56:45,574 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:45,574 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:45,574 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:45,574 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-19 07:56:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:45,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42359,1681890819191' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-19 07:56:45,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70cca534[Count = 0] remaining members to acquire global barrier 2023-04-19 07:56:45,575 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-19 07:56:45,575 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:45,576 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:45,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:45,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:45,576 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-19 07:56:45,576 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-19 07:56:45,576 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42359,1681890819191' in zk 2023-04-19 07:56:45,576 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,576 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-19 07:56:45,577 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-19 07:56:45,577 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,577 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:45,578 DEBUG [member: 'jenkins-hbase4.apache.org,42359,1681890819191' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-19 07:56:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42359,1681890819191': 2023-04-19 07:56:45,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42359,1681890819191' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-19 07:56:45,580 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-19 07:56:45,580 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-19 07:56:45,580 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-19 07:56:45,580 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-19 07:56:45,580 INFO [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-19 07:56:45,582 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:45,582 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-19 07:56:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-19 07:56:45,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:45,582 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:45,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-19 07:56:45,582 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-19 07:56:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:45,583 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-19 07:56:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-19 07:56:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:45,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-19 07:56:45,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-19 07:56:45,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-19 07:56:45,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-19 07:56:45,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-19 07:56:45,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:45,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-19 07:56:45,586 DEBUG [Thread-3718] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-19 07:56:45,586 DEBUG [Thread-3718] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-19 07:56:45,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-19 07:56:45,585 DEBUG [(jenkins-hbase4.apache.org,43905,1681890817995)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-19 07:56:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-19 07:56:45,585 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:45,586 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-19 07:56:45,586 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-19 07:56:45,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-19 07:56:45,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-19 07:56:45,686 DEBUG [Thread-3718] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-19 07:56:45,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-19 07:56:46,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42359] regionserver.HRegion(9158): Flush requested on 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:46,905 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=87.22 KB heapSize=243.84 KB 2023-04-19 07:56:46,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:46,905 DEBUG [Thread-3708] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d9fccbf to 127.0.0.1:58087 2023-04-19 07:56:46,905 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:46,905 DEBUG [Thread-3708] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:46,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:46,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:46,906 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:46,906 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-19 07:56:46,920 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419aa86fe342465403ab3c0c6fe47a23c7b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419aa86fe342465403ab3c0c6fe47a23c7b 2023-04-19 07:56:46,935 DEBUG [Thread-3712] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x03ced4ba to 127.0.0.1:58087 2023-04-19 07:56:46,935 DEBUG [Thread-3712] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:46,950 DEBUG [Thread-3716] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59185ba7 to 127.0.0.1:58087 2023-04-19 07:56:46,950 DEBUG [Thread-3716] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:46,969 DEBUG [Thread-3714] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63da10ca to 127.0.0.1:58087 2023-04-19 07:56:46,969 DEBUG [Thread-3714] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:46,969 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-19 07:56:46,970 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 32 2023-04-19 07:56:46,970 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 418 2023-04-19 07:56:46,970 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 38 2023-04-19 07:56:46,970 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 35 2023-04-19 07:56:46,970 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(394): wrote 35 2023-04-19 07:56:46,970 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-19 07:56:46,970 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(398): read 4220 2023-04-19 07:56:46,970 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(398): read 4077 2023-04-19 07:56:46,970 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(398): read 4171 2023-04-19 07:56:46,970 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(398): read 4219 2023-04-19 07:56:46,970 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(398): read 4076 2023-04-19 07:56:46,970 INFO [Listener at localhost/42517] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-19 07:56:46,970 INFO [Listener at localhost/42517] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-19 07:56:46,970 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a9f3972 to 127.0.0.1:58087 2023-04-19 07:56:46,970 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:46,971 INFO [Listener at localhost/42517] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-19 07:56:46,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-19 07:56:46,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:46,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-19 07:56:46,974 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681891006974"}]},"ts":"1681891006974"} 2023-04-19 07:56:46,975 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-19 07:56:46,976 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-19 07:56:46,977 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=051c522ed965d63912b88bb381b9d1d2, UNASSIGN}] 2023-04-19 07:56:46,978 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=051c522ed965d63912b88bb381b9d1d2, UNASSIGN 2023-04-19 07:56:46,978 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=051c522ed965d63912b88bb381b9d1d2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:46,978 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681891006978"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681891006978"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681891006978"}]},"ts":"1681891006978"} 2023-04-19 07:56:46,979 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191}] 2023-04-19 07:56:47,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-19 07:56:47,131 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:47,131 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 051c522ed965d63912b88bb381b9d1d2, disabling compactions & flushes 2023-04-19 07:56:47,131 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:47,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-19 07:56:47,325 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=668, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/ee2fa43293254f32baf118dfa554c77d 2023-04-19 07:56:47,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-19 07:56:47,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/3a88acd14f5a4cba8d3aad92f81af553 2023-04-19 07:56:48,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-19 07:56:48,143 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/e104e091f9c2433283db52c367a2c280 2023-04-19 07:56:48,146 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/ee2fa43293254f32baf118dfa554c77d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ee2fa43293254f32baf118dfa554c77d 2023-04-19 07:56:48,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ee2fa43293254f32baf118dfa554c77d, entries=150, sequenceid=668, filesize=25.4 K 2023-04-19 07:56:48,150 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/3a88acd14f5a4cba8d3aad92f81af553 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/3a88acd14f5a4cba8d3aad92f81af553 2023-04-19 07:56:48,153 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/3a88acd14f5a4cba8d3aad92f81af553, entries=150, sequenceid=668, filesize=11.9 K 2023-04-19 07:56:48,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/e104e091f9c2433283db52c367a2c280 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e104e091f9c2433283db52c367a2c280 2023-04-19 07:56:48,157 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e104e091f9c2433283db52c367a2c280, entries=150, sequenceid=668, filesize=11.9 K 2023-04-19 07:56:48,157 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~243.80 KB/249648, currentSize=20.13 KB/20610 for 051c522ed965d63912b88bb381b9d1d2 in 1252ms, sequenceid=668, compaction requested=true 2023-04-19 07:56:48,158 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:48,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:48,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:48,158 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. because compaction request was cancelled 2023-04-19 07:56:48,158 DEBUG [RS:0;jenkins-hbase4:42359-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. because compaction request was cancelled 2023-04-19 07:56:48,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-19 07:56:48,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:48,158 DEBUG [RS:0;jenkins-hbase4:42359-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. because compaction request was cancelled 2023-04-19 07:56:48,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-19 07:56:48,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. after waiting 0 ms 2023-04-19 07:56:48,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:48,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 051c522ed965d63912b88bb381b9d1d2 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-19 07:56:48,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=A 2023-04-19 07:56:48,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:48,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=B 2023-04-19 07:56:48,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:48,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 051c522ed965d63912b88bb381b9d1d2, store=C 2023-04-19 07:56:48,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-19 07:56:48,569 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230419d2c611c38ac6490b80b097975fe17b18 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d2c611c38ac6490b80b097975fe17b18 2023-04-19 07:56:48,974 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=674, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/36cd7342ad464bb89d803083d2b74f54 2023-04-19 07:56:49,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-19 07:56:49,382 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/b815c70f98b744aebc83ac7dd9b35452 2023-04-19 07:56:49,791 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=674 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/996c6c980b41480c84899d15e4067c80 2023-04-19 07:56:49,794 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/A/36cd7342ad464bb89d803083d2b74f54 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/36cd7342ad464bb89d803083d2b74f54 2023-04-19 07:56:49,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/36cd7342ad464bb89d803083d2b74f54, entries=150, sequenceid=674, filesize=25.4 K 2023-04-19 07:56:49,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/B/b815c70f98b744aebc83ac7dd9b35452 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/b815c70f98b744aebc83ac7dd9b35452 2023-04-19 07:56:49,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/b815c70f98b744aebc83ac7dd9b35452, entries=150, sequenceid=674, filesize=11.9 K 2023-04-19 07:56:49,801 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/.tmp/C/996c6c980b41480c84899d15e4067c80 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/996c6c980b41480c84899d15e4067c80 2023-04-19 07:56:49,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/996c6c980b41480c84899d15e4067c80, entries=150, sequenceid=674, filesize=11.9 K 2023-04-19 07:56:49,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 051c522ed965d63912b88bb381b9d1d2 in 1646ms, sequenceid=674, compaction requested=true 2023-04-19 07:56:49,805 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/7c41184935594c28a5b080c2250534ff, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/9e1b51b6031f413aa0a68f92816b7618, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/8fbfc346eadb4648948e701719685988, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/5f4b234f704f4a6ca40eb79d503d250f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/68af14c5a04748deb6f35c23028124d5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/61dcfacfd4a841499aad606971c9d258, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/10b12855608d434d8195ed48fe7ffc96, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/f60f9baca9dc444caf34f9dad2c83e8a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/fd4be2d75b834cd69f83da9710117f45, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/fee620c5b44b49ff8fb011b2889dd985, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/14aac196c6694543a1417f6604ce4104, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/a8ce5fc26b7a454da77cb4054b09d1a8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/2bd5ae90fc044556a55a3ab2b8ed41da, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/67f11c86567f41d2ae4290566fa6b90f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/56bf700403aa4dd3ba5c885150e3a4a8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/98d84c6d1bd7466686f7a928e3b2e7e9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/3a2baedd34c743bcba794599f0d49e6c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/d91851fd4011453fb0ecbd6b39533abf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/cbdb5db2e4cf4c7581c531f5dae75716, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/6cdb3ee83c954680b0cd45886a6b0d6a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/eb68b9cf996142ee81c04a009c224a0e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ddd06ab0484e4dc59ee67cd1b5b8f186, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/16ecb23236ac496b996f2868783d6c99, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/047e4174ea864945ae494fb11bffc1d5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/06e0cb2a06d04970a52833f542d669bb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/8ac9d4036c864b9db901f67415437d26, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ad51e73c10574cd6ad40efabf3954cbf, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/79be28dd10ff45b68fa76f195cabee2b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/c6f106f950824707b34f2245cafcaafd, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/461baeca889e45b5b7ca4244dbe70873, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/f3e49958f97e401c906df7b4e4a42d89] to archive 2023-04-19 07:56:49,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:56:49,807 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/7c41184935594c28a5b080c2250534ff to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/7c41184935594c28a5b080c2250534ff 2023-04-19 07:56:49,808 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/9e1b51b6031f413aa0a68f92816b7618 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/9e1b51b6031f413aa0a68f92816b7618 2023-04-19 07:56:49,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/8fbfc346eadb4648948e701719685988 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/8fbfc346eadb4648948e701719685988 2023-04-19 07:56:49,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/5f4b234f704f4a6ca40eb79d503d250f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/5f4b234f704f4a6ca40eb79d503d250f 2023-04-19 07:56:49,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/68af14c5a04748deb6f35c23028124d5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/68af14c5a04748deb6f35c23028124d5 2023-04-19 07:56:49,811 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/61dcfacfd4a841499aad606971c9d258 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/61dcfacfd4a841499aad606971c9d258 2023-04-19 07:56:49,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/10b12855608d434d8195ed48fe7ffc96 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/10b12855608d434d8195ed48fe7ffc96 2023-04-19 07:56:49,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/f60f9baca9dc444caf34f9dad2c83e8a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/f60f9baca9dc444caf34f9dad2c83e8a 2023-04-19 07:56:49,813 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/fd4be2d75b834cd69f83da9710117f45 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/fd4be2d75b834cd69f83da9710117f45 2023-04-19 07:56:49,814 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/fee620c5b44b49ff8fb011b2889dd985 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/fee620c5b44b49ff8fb011b2889dd985 2023-04-19 07:56:49,815 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/14aac196c6694543a1417f6604ce4104 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/14aac196c6694543a1417f6604ce4104 2023-04-19 07:56:49,815 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/a8ce5fc26b7a454da77cb4054b09d1a8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/a8ce5fc26b7a454da77cb4054b09d1a8 2023-04-19 07:56:49,816 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/2bd5ae90fc044556a55a3ab2b8ed41da to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/2bd5ae90fc044556a55a3ab2b8ed41da 2023-04-19 07:56:49,817 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/67f11c86567f41d2ae4290566fa6b90f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/67f11c86567f41d2ae4290566fa6b90f 2023-04-19 07:56:49,817 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/56bf700403aa4dd3ba5c885150e3a4a8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/56bf700403aa4dd3ba5c885150e3a4a8 2023-04-19 07:56:49,818 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/98d84c6d1bd7466686f7a928e3b2e7e9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/98d84c6d1bd7466686f7a928e3b2e7e9 2023-04-19 07:56:49,819 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/3a2baedd34c743bcba794599f0d49e6c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/3a2baedd34c743bcba794599f0d49e6c 2023-04-19 07:56:49,820 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/d91851fd4011453fb0ecbd6b39533abf to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/d91851fd4011453fb0ecbd6b39533abf 2023-04-19 07:56:49,820 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/cbdb5db2e4cf4c7581c531f5dae75716 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/cbdb5db2e4cf4c7581c531f5dae75716 2023-04-19 07:56:49,821 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/6cdb3ee83c954680b0cd45886a6b0d6a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/6cdb3ee83c954680b0cd45886a6b0d6a 2023-04-19 07:56:49,822 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/eb68b9cf996142ee81c04a009c224a0e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/eb68b9cf996142ee81c04a009c224a0e 2023-04-19 07:56:49,822 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ddd06ab0484e4dc59ee67cd1b5b8f186 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ddd06ab0484e4dc59ee67cd1b5b8f186 2023-04-19 07:56:49,823 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/16ecb23236ac496b996f2868783d6c99 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/16ecb23236ac496b996f2868783d6c99 2023-04-19 07:56:49,824 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/047e4174ea864945ae494fb11bffc1d5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/047e4174ea864945ae494fb11bffc1d5 2023-04-19 07:56:49,825 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/06e0cb2a06d04970a52833f542d669bb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/06e0cb2a06d04970a52833f542d669bb 2023-04-19 07:56:49,825 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/8ac9d4036c864b9db901f67415437d26 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/8ac9d4036c864b9db901f67415437d26 2023-04-19 07:56:49,826 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ad51e73c10574cd6ad40efabf3954cbf to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ad51e73c10574cd6ad40efabf3954cbf 2023-04-19 07:56:49,827 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/79be28dd10ff45b68fa76f195cabee2b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/79be28dd10ff45b68fa76f195cabee2b 2023-04-19 07:56:49,828 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/c6f106f950824707b34f2245cafcaafd to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/c6f106f950824707b34f2245cafcaafd 2023-04-19 07:56:49,829 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/461baeca889e45b5b7ca4244dbe70873 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/461baeca889e45b5b7ca4244dbe70873 2023-04-19 07:56:49,829 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/f3e49958f97e401c906df7b4e4a42d89 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/f3e49958f97e401c906df7b4e4a42d89 2023-04-19 07:56:49,832 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4e7009dc4f0840c6b2e2f9d3f3308ccc, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/16efe36d98e147f1a596e6bec27ada5a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/74dabad365af4b0dbd8118e3155541a8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4f59fd14cdd542fc91392e8ffd17d482, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9af8c2e161a141af964e163a4ab9ea30, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/a8256069426649ad9bd5be0cb9a9256c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/10cfcfd8fe4a4fd495ba7e94bdfa80df, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1681dc94d782439eaea2e08b2b97ec41, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1f165f673a4e4b6d8ab1d56f8b898b44, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9c4f36467a944fd8bd565d625f1313e3, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/171fa50402ff4afe88cb9f368cb531f1, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/10602b28edb94137bb77b571a6becad1, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1b547ada454347e5bea848cea2f0744e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ee1faafe31a74cee939012d80445f521, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/db55488ce52f487397deb8c61da7b24e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/a715d6c142b842c7a6540fab5bcb378c, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/262ecd5001f34e07aa42f72fa646a464, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/e1458116ab42486fa82f77bf444ce7c5, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/bde49fe3bbff4810917d81d3badffcfa, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/7ef114e792634d5684947866c4594c4e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/2f8c587766744d87871905cf339db55b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ef5dd9699cdc452d9aa08799ab18f99d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ef52e2b245b64b20a8b32ca4012ac55f, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/844cb4024b5445008a408a37dc738167, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/e87deb90cc4f48e48685f78e02160e57, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/557eac5ae91d45029cd3c708e72752a4, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/27309727138f4b8695017b354c082ebb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/5f8d0d0c02ec433b864f82ccb237f3bb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4e4ba1e6f23c4085a42c169f38ad02d7, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/37b712ce85224f90bf059bf0a31b7ab9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9485887385174e65812ba4db89c73853] to archive 2023-04-19 07:56:49,833 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:56:49,834 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4e7009dc4f0840c6b2e2f9d3f3308ccc to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4e7009dc4f0840c6b2e2f9d3f3308ccc 2023-04-19 07:56:49,835 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/16efe36d98e147f1a596e6bec27ada5a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/16efe36d98e147f1a596e6bec27ada5a 2023-04-19 07:56:49,836 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/74dabad365af4b0dbd8118e3155541a8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/74dabad365af4b0dbd8118e3155541a8 2023-04-19 07:56:49,836 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4f59fd14cdd542fc91392e8ffd17d482 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4f59fd14cdd542fc91392e8ffd17d482 2023-04-19 07:56:49,837 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9af8c2e161a141af964e163a4ab9ea30 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9af8c2e161a141af964e163a4ab9ea30 2023-04-19 07:56:49,838 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/a8256069426649ad9bd5be0cb9a9256c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/a8256069426649ad9bd5be0cb9a9256c 2023-04-19 07:56:49,839 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/10cfcfd8fe4a4fd495ba7e94bdfa80df to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/10cfcfd8fe4a4fd495ba7e94bdfa80df 2023-04-19 07:56:49,840 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1681dc94d782439eaea2e08b2b97ec41 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1681dc94d782439eaea2e08b2b97ec41 2023-04-19 07:56:49,840 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1f165f673a4e4b6d8ab1d56f8b898b44 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1f165f673a4e4b6d8ab1d56f8b898b44 2023-04-19 07:56:49,841 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9c4f36467a944fd8bd565d625f1313e3 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9c4f36467a944fd8bd565d625f1313e3 2023-04-19 07:56:49,842 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/171fa50402ff4afe88cb9f368cb531f1 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/171fa50402ff4afe88cb9f368cb531f1 2023-04-19 07:56:49,843 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/10602b28edb94137bb77b571a6becad1 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/10602b28edb94137bb77b571a6becad1 2023-04-19 07:56:49,844 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1b547ada454347e5bea848cea2f0744e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/1b547ada454347e5bea848cea2f0744e 2023-04-19 07:56:49,845 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ee1faafe31a74cee939012d80445f521 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ee1faafe31a74cee939012d80445f521 2023-04-19 07:56:49,846 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/db55488ce52f487397deb8c61da7b24e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/db55488ce52f487397deb8c61da7b24e 2023-04-19 07:56:49,846 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/a715d6c142b842c7a6540fab5bcb378c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/a715d6c142b842c7a6540fab5bcb378c 2023-04-19 07:56:49,847 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/262ecd5001f34e07aa42f72fa646a464 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/262ecd5001f34e07aa42f72fa646a464 2023-04-19 07:56:49,848 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/e1458116ab42486fa82f77bf444ce7c5 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/e1458116ab42486fa82f77bf444ce7c5 2023-04-19 07:56:49,849 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/bde49fe3bbff4810917d81d3badffcfa to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/bde49fe3bbff4810917d81d3badffcfa 2023-04-19 07:56:49,850 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/7ef114e792634d5684947866c4594c4e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/7ef114e792634d5684947866c4594c4e 2023-04-19 07:56:49,850 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/2f8c587766744d87871905cf339db55b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/2f8c587766744d87871905cf339db55b 2023-04-19 07:56:49,851 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ef5dd9699cdc452d9aa08799ab18f99d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ef5dd9699cdc452d9aa08799ab18f99d 2023-04-19 07:56:49,852 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ef52e2b245b64b20a8b32ca4012ac55f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ef52e2b245b64b20a8b32ca4012ac55f 2023-04-19 07:56:49,853 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/844cb4024b5445008a408a37dc738167 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/844cb4024b5445008a408a37dc738167 2023-04-19 07:56:49,854 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/e87deb90cc4f48e48685f78e02160e57 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/e87deb90cc4f48e48685f78e02160e57 2023-04-19 07:56:49,854 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/557eac5ae91d45029cd3c708e72752a4 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/557eac5ae91d45029cd3c708e72752a4 2023-04-19 07:56:49,855 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/27309727138f4b8695017b354c082ebb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/27309727138f4b8695017b354c082ebb 2023-04-19 07:56:49,857 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/5f8d0d0c02ec433b864f82ccb237f3bb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/5f8d0d0c02ec433b864f82ccb237f3bb 2023-04-19 07:56:49,857 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4e4ba1e6f23c4085a42c169f38ad02d7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/4e4ba1e6f23c4085a42c169f38ad02d7 2023-04-19 07:56:49,858 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/37b712ce85224f90bf059bf0a31b7ab9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/37b712ce85224f90bf059bf0a31b7ab9 2023-04-19 07:56:49,859 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9485887385174e65812ba4db89c73853 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/9485887385174e65812ba4db89c73853 2023-04-19 07:56:49,862 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5380f65373eb4e07999569a56cc9057b, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/3ff0968a9f394aabab0e32ca0d727815, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/f12abd94895e4d31b0014b7e4131653a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/beb609c6a6234e73bf274807ebaeaf87, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/261e17d077c546358c96151c42782348, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/1d0d01f3e10f44a2bce2def1d0c05a88, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/ca853285a6204f7997e3e2cbe8047737, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/b0242ec4007f404da580cb27ac27f26a, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e2d18907838a4f9d87b5bb0fb02fac35, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/8a037f11aa4d47a89c7427116806156d, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/0839ad88c826439fb918cd40f11d62fb, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/8eb76521429f42e5b8dfad0226af6ae3, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/b1eba3a5f5bc4fe1bcbf20d3fa522694, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/a1692cdd1e0e4fbcbe338066db255461, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/2959a99a08e44ac190914fd373fd5b59, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/cf473e537bad420598b81c058c48d831, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/da445a01280d475fafd06b7d1ca4ff74, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/9b13fdbabb8c457fbb7d4e27c2f58881, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5750e6e7ddd849d8ba1655503427b5c0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5684eb70047140e3b86ab598e6545608, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/0e466ac215544f939020430488ebb1e0, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e6255424d1bf4de4969f3b55385f95c9, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5dc86d0b7b9c4e479f63cd1752d7a94e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e2ee022827d24504a01b09b899fb14f8, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e264db1687814b5293c572d90daf3a5e, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/492ec3de141246feb1c4a146815d5837, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/7262aa52217e4a9ca1982d4f849ccf15, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/2270fdd017cf4df6a605d287f05320ea, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/cb809e82e1394ca58be21d39411a6b29, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e61bbedf3a964519912780dd4d79b4bb] to archive 2023-04-19 07:56:49,862 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-19 07:56:49,863 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5380f65373eb4e07999569a56cc9057b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5380f65373eb4e07999569a56cc9057b 2023-04-19 07:56:49,864 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/3ff0968a9f394aabab0e32ca0d727815 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/3ff0968a9f394aabab0e32ca0d727815 2023-04-19 07:56:49,865 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/f12abd94895e4d31b0014b7e4131653a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/f12abd94895e4d31b0014b7e4131653a 2023-04-19 07:56:49,866 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/beb609c6a6234e73bf274807ebaeaf87 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/beb609c6a6234e73bf274807ebaeaf87 2023-04-19 07:56:49,867 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/261e17d077c546358c96151c42782348 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/261e17d077c546358c96151c42782348 2023-04-19 07:56:49,867 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/1d0d01f3e10f44a2bce2def1d0c05a88 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/1d0d01f3e10f44a2bce2def1d0c05a88 2023-04-19 07:56:49,868 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/ca853285a6204f7997e3e2cbe8047737 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/ca853285a6204f7997e3e2cbe8047737 2023-04-19 07:56:49,869 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/b0242ec4007f404da580cb27ac27f26a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/b0242ec4007f404da580cb27ac27f26a 2023-04-19 07:56:49,870 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e2d18907838a4f9d87b5bb0fb02fac35 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e2d18907838a4f9d87b5bb0fb02fac35 2023-04-19 07:56:49,870 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/8a037f11aa4d47a89c7427116806156d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/8a037f11aa4d47a89c7427116806156d 2023-04-19 07:56:49,871 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/0839ad88c826439fb918cd40f11d62fb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/0839ad88c826439fb918cd40f11d62fb 2023-04-19 07:56:49,872 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/8eb76521429f42e5b8dfad0226af6ae3 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/8eb76521429f42e5b8dfad0226af6ae3 2023-04-19 07:56:49,873 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/b1eba3a5f5bc4fe1bcbf20d3fa522694 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/b1eba3a5f5bc4fe1bcbf20d3fa522694 2023-04-19 07:56:49,873 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/a1692cdd1e0e4fbcbe338066db255461 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/a1692cdd1e0e4fbcbe338066db255461 2023-04-19 07:56:49,874 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/2959a99a08e44ac190914fd373fd5b59 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/2959a99a08e44ac190914fd373fd5b59 2023-04-19 07:56:49,875 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/cf473e537bad420598b81c058c48d831 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/cf473e537bad420598b81c058c48d831 2023-04-19 07:56:49,876 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/da445a01280d475fafd06b7d1ca4ff74 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/da445a01280d475fafd06b7d1ca4ff74 2023-04-19 07:56:49,876 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/9b13fdbabb8c457fbb7d4e27c2f58881 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/9b13fdbabb8c457fbb7d4e27c2f58881 2023-04-19 07:56:49,877 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5750e6e7ddd849d8ba1655503427b5c0 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5750e6e7ddd849d8ba1655503427b5c0 2023-04-19 07:56:49,878 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5684eb70047140e3b86ab598e6545608 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5684eb70047140e3b86ab598e6545608 2023-04-19 07:56:49,879 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/0e466ac215544f939020430488ebb1e0 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/0e466ac215544f939020430488ebb1e0 2023-04-19 07:56:49,880 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e6255424d1bf4de4969f3b55385f95c9 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e6255424d1bf4de4969f3b55385f95c9 2023-04-19 07:56:49,881 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5dc86d0b7b9c4e479f63cd1752d7a94e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/5dc86d0b7b9c4e479f63cd1752d7a94e 2023-04-19 07:56:49,882 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e2ee022827d24504a01b09b899fb14f8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e2ee022827d24504a01b09b899fb14f8 2023-04-19 07:56:49,882 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e264db1687814b5293c572d90daf3a5e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e264db1687814b5293c572d90daf3a5e 2023-04-19 07:56:49,883 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/492ec3de141246feb1c4a146815d5837 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/492ec3de141246feb1c4a146815d5837 2023-04-19 07:56:49,884 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/7262aa52217e4a9ca1982d4f849ccf15 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/7262aa52217e4a9ca1982d4f849ccf15 2023-04-19 07:56:49,885 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/2270fdd017cf4df6a605d287f05320ea to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/2270fdd017cf4df6a605d287f05320ea 2023-04-19 07:56:49,886 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/cb809e82e1394ca58be21d39411a6b29 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/cb809e82e1394ca58be21d39411a6b29 2023-04-19 07:56:49,887 DEBUG [StoreCloser-TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e61bbedf3a964519912780dd4d79b4bb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e61bbedf3a964519912780dd4d79b4bb 2023-04-19 07:56:49,891 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/recovered.edits/677.seqid, newMaxSeqId=677, maxSeqId=4 2023-04-19 07:56:49,892 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2. 2023-04-19 07:56:49,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 051c522ed965d63912b88bb381b9d1d2: 2023-04-19 07:56:49,893 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:49,894 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=051c522ed965d63912b88bb381b9d1d2, regionState=CLOSED 2023-04-19 07:56:49,894 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681891009894"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681891009894"}]},"ts":"1681891009894"} 2023-04-19 07:56:49,896 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-19 07:56:49,896 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 051c522ed965d63912b88bb381b9d1d2, server=jenkins-hbase4.apache.org,42359,1681890819191 in 2.9160 sec 2023-04-19 07:56:49,897 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-19 07:56:49,897 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=051c522ed965d63912b88bb381b9d1d2, UNASSIGN in 2.9190 sec 2023-04-19 07:56:49,897 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681891009897"}]},"ts":"1681891009897"} 2023-04-19 07:56:49,898 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-19 07:56:49,900 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-19 07:56:49,901 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.9290 sec 2023-04-19 07:56:51,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-19 07:56:51,079 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-19 07:56:51,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-19 07:56:51,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:51,082 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:51,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-19 07:56:51,083 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:51,087 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:51,089 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C, FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/recovered.edits] 2023-04-19 07:56:51,091 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/36cd7342ad464bb89d803083d2b74f54 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/36cd7342ad464bb89d803083d2b74f54 2023-04-19 07:56:51,092 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/d62e577018fc465780e7a7d088d6ee39 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/d62e577018fc465780e7a7d088d6ee39 2023-04-19 07:56:51,094 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/da70f5aec25b4702865ac7daf1c3ec66 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/da70f5aec25b4702865ac7daf1c3ec66 2023-04-19 07:56:51,095 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ee2fa43293254f32baf118dfa554c77d to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/A/ee2fa43293254f32baf118dfa554c77d 2023-04-19 07:56:51,096 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/2f3439ed71f241d69d0e2b9833729419 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/2f3439ed71f241d69d0e2b9833729419 2023-04-19 07:56:51,097 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/3a88acd14f5a4cba8d3aad92f81af553 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/3a88acd14f5a4cba8d3aad92f81af553 2023-04-19 07:56:51,098 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/b815c70f98b744aebc83ac7dd9b35452 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/b815c70f98b744aebc83ac7dd9b35452 2023-04-19 07:56:51,099 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ba5597d18edb456fa754e612ea2a063a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/B/ba5597d18edb456fa754e612ea2a063a 2023-04-19 07:56:51,100 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/6988419e14854218ba908dd0a7df8114 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/6988419e14854218ba908dd0a7df8114 2023-04-19 07:56:51,101 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/996c6c980b41480c84899d15e4067c80 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/996c6c980b41480c84899d15e4067c80 2023-04-19 07:56:51,102 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e104e091f9c2433283db52c367a2c280 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e104e091f9c2433283db52c367a2c280 2023-04-19 07:56:51,102 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e78c756d805b4f0ab77d49b1316f1044 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/C/e78c756d805b4f0ab77d49b1316f1044 2023-04-19 07:56:51,104 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/recovered.edits/677.seqid to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2/recovered.edits/677.seqid 2023-04-19 07:56:51,104 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/.tmp/data/default/TestAcidGuarantees/051c522ed965d63912b88bb381b9d1d2 2023-04-19 07:56:51,104 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-19 07:56:51,105 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-19 07:56:51,105 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-19 07:56:51,107 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041904fdaef7d0324b97aeb4afb1e27023aa to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041904fdaef7d0324b97aeb4afb1e27023aa 2023-04-19 07:56:51,108 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191392853bab9a4798ba89ed06beec704a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304191392853bab9a4798ba89ed06beec704a 2023-04-19 07:56:51,109 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041917e5ea676bc548d58583aee8bc0e6145 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041917e5ea676bc548d58583aee8bc0e6145 2023-04-19 07:56:51,110 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041919d79781b3c64e27bd9efc82d674d41c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041919d79781b3c64e27bd9efc82d674d41c 2023-04-19 07:56:51,110 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041926614aefca8b4a1db2958228699f5195 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041926614aefca8b4a1db2958228699f5195 2023-04-19 07:56:51,111 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192a61bfb5a6a844d98ba8cc1ca3813da7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192a61bfb5a6a844d98ba8cc1ca3813da7 2023-04-19 07:56:51,112 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192e35ef8b31c44a32ae1448c608bddef3 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304192e35ef8b31c44a32ae1448c608bddef3 2023-04-19 07:56:51,113 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041941de455fcfec4ee8a00b6540fe933322 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041941de455fcfec4ee8a00b6540fe933322 2023-04-19 07:56:51,114 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194b4e31fd28a24a19a9dda8deec19338f to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304194b4e31fd28a24a19a9dda8deec19338f 2023-04-19 07:56:51,115 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195889e10f3b264b5a863d2a2b307a029c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304195889e10f3b264b5a863d2a2b307a029c 2023-04-19 07:56:51,116 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197275298f1e2641ae87359aa0b6188fc8 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197275298f1e2641ae87359aa0b6188fc8 2023-04-19 07:56:51,117 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041975fad4e26c404e4db3074eb94fb86021 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041975fad4e26c404e4db3074eb94fb86021 2023-04-19 07:56:51,118 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197d921f971ed7496ea7d458e2769de2fb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304197d921f971ed7496ea7d458e2769de2fb 2023-04-19 07:56:51,118 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041981e5cb3eebf6419ca6907287ee6f73bf to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041981e5cb3eebf6419ca6907287ee6f73bf 2023-04-19 07:56:51,119 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041991782e7f6581408cbafa2590cd29a15c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041991782e7f6581408cbafa2590cd29a15c 2023-04-19 07:56:51,120 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199cc38301bf594494b897891411630950 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199cc38301bf594494b897891411630950 2023-04-19 07:56:51,120 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199f25178c314e4d9da5615e98c16b1fb7 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199f25178c314e4d9da5615e98c16b1fb7 2023-04-19 07:56:51,121 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199f5c1dd497ad415aafc1611c0913f09a to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304199f5c1dd497ad415aafc1611c0913f09a 2023-04-19 07:56:51,122 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419aa86fe342465403ab3c0c6fe47a23c7b to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419aa86fe342465403ab3c0c6fe47a23c7b 2023-04-19 07:56:51,123 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b8791152d2354d9092d0da98d9af8592 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419b8791152d2354d9092d0da98d9af8592 2023-04-19 07:56:51,123 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d2c611c38ac6490b80b097975fe17b18 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419d2c611c38ac6490b80b097975fe17b18 2023-04-19 07:56:51,124 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e6828bc897ed4f96b4cb00fe2bf55572 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e6828bc897ed4f96b4cb00fe2bf55572 2023-04-19 07:56:51,125 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e86a778957ab44c1a6d6b899590ceab1 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419e86a778957ab44c1a6d6b899590ceab1 2023-04-19 07:56:51,126 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419eb6bd23a70f8444e9f0d76e666bb37eb to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419eb6bd23a70f8444e9f0d76e666bb37eb 2023-04-19 07:56:51,126 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f4a05e6b480b4dbeb726a7b3b3d93f0e to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f4a05e6b480b4dbeb726a7b3b3d93f0e 2023-04-19 07:56:51,127 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f73e43d74f8745ebb90a7cbd328d277c to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f73e43d74f8745ebb90a7cbd328d277c 2023-04-19 07:56:51,128 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f9619c20265f4516b83d3998b372eb79 to hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230419f9619c20265f4516b83d3998b372eb79 2023-04-19 07:56:51,128 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-19 07:56:51,130 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:51,131 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-19 07:56:51,133 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-19 07:56:51,133 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:51,133 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-19 07:56:51,134 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681891011133"}]},"ts":"9223372036854775807"} 2023-04-19 07:56:51,135 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-19 07:56:51,135 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 051c522ed965d63912b88bb381b9d1d2, NAME => 'TestAcidGuarantees,,1681890982968.051c522ed965d63912b88bb381b9d1d2.', STARTKEY => '', ENDKEY => ''}] 2023-04-19 07:56:51,135 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-19 07:56:51,135 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681891011135"}]},"ts":"9223372036854775807"} 2023-04-19 07:56:51,136 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-19 07:56:51,139 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-19 07:56:51,139 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 59 msec 2023-04-19 07:56:51,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-19 07:56:51,184 INFO [Listener at localhost/42517] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-19 07:56:51,196 INFO [Listener at localhost/42517] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 227) - Thread LEAK? -, OpenFileDescriptor=560 (was 553) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=430 (was 450), ProcessCount=169 (was 169), AvailableMemoryMB=6683 (was 6649) - AvailableMemoryMB LEAK? - 2023-04-19 07:56:51,196 INFO [Listener at localhost/42517] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-19 07:56:51,196 INFO [Listener at localhost/42517] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-19 07:56:51,197 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ddd91d3 to 127.0.0.1:58087 2023-04-19 07:56:51,197 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:51,197 DEBUG [Listener at localhost/42517] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-19 07:56:51,197 DEBUG [Listener at localhost/42517] util.JVMClusterUtil(257): Found active master hash=1964207311, stopped=false 2023-04-19 07:56:51,197 INFO [Listener at localhost/42517] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,43905,1681890817995 2023-04-19 07:56:51,199 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-19 07:56:51,199 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-19 07:56:51,199 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 07:56:51,199 INFO [Listener at localhost/42517] procedure2.ProcedureExecutor(629): Stopping 2023-04-19 07:56:51,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-19 07:56:51,199 DEBUG [Listener at localhost/42517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x162b1133 to 127.0.0.1:58087 2023-04-19 07:56:51,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-19 07:56:51,199 DEBUG [Listener at localhost/42517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:51,200 INFO [Listener at localhost/42517] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42359,1681890819191' ***** 2023-04-19 07:56:51,200 INFO [Listener at localhost/42517] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-19 07:56:51,200 INFO [RS:0;jenkins-hbase4:42359] regionserver.HeapMemoryManager(220): Stopping 2023-04-19 07:56:51,200 INFO [RS:0;jenkins-hbase4:42359] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-19 07:56:51,200 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-19 07:56:51,200 INFO [RS:0;jenkins-hbase4:42359] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-19 07:56:51,200 INFO [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(3304): Received CLOSE for 6189702b8073b2778b13e3b2b656c082 2023-04-19 07:56:51,201 INFO [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:51,201 DEBUG [RS:0;jenkins-hbase4:42359] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25146046 to 127.0.0.1:58087 2023-04-19 07:56:51,201 DEBUG [RS:0;jenkins-hbase4:42359] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:51,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6189702b8073b2778b13e3b2b656c082, disabling compactions & flushes 2023-04-19 07:56:51,202 INFO [RS:0;jenkins-hbase4:42359] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-19 07:56:51,202 INFO [RS:0;jenkins-hbase4:42359] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-19 07:56:51,202 INFO [RS:0;jenkins-hbase4:42359] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-19 07:56:51,202 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082. 2023-04-19 07:56:51,202 INFO [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-19 07:56:51,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082. 2023-04-19 07:56:51,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082. after waiting 0 ms 2023-04-19 07:56:51,202 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082. 2023-04-19 07:56:51,202 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 6189702b8073b2778b13e3b2b656c082 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-19 07:56:51,202 INFO [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-19 07:56:51,202 DEBUG [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 6189702b8073b2778b13e3b2b656c082=hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082.} 2023-04-19 07:56:51,203 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-19 07:56:51,203 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-19 07:56:51,203 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-19 07:56:51,203 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-19 07:56:51,203 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-19 07:56:51,203 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-19 07:56:51,204 DEBUG [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1505): Waiting on 1588230740, 6189702b8073b2778b13e3b2b656c082 2023-04-19 07:56:51,267 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-19 07:56:51,405 DEBUG [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1505): Waiting on 1588230740, 6189702b8073b2778b13e3b2b656c082 2023-04-19 07:56:51,605 DEBUG [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1505): Waiting on 1588230740, 6189702b8073b2778b13e3b2b656c082 2023-04-19 07:56:51,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/namespace/6189702b8073b2778b13e3b2b656c082/.tmp/info/78f29fd97dde425e8c59820394756c53 2023-04-19 07:56:51,614 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/.tmp/info/7619c3a991d044258f65252693cb030d 2023-04-19 07:56:51,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/namespace/6189702b8073b2778b13e3b2b656c082/.tmp/info/78f29fd97dde425e8c59820394756c53 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/namespace/6189702b8073b2778b13e3b2b656c082/info/78f29fd97dde425e8c59820394756c53 2023-04-19 07:56:51,618 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7619c3a991d044258f65252693cb030d 2023-04-19 07:56:51,621 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/namespace/6189702b8073b2778b13e3b2b656c082/info/78f29fd97dde425e8c59820394756c53, entries=2, sequenceid=6, filesize=4.8 K 2023-04-19 07:56:51,622 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 6189702b8073b2778b13e3b2b656c082 in 420ms, sequenceid=6, compaction requested=false 2023-04-19 07:56:51,624 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/.tmp/rep_barrier/a96990349c624212b55fb1d571d42f5e 2023-04-19 07:56:51,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/namespace/6189702b8073b2778b13e3b2b656c082/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-19 07:56:51,627 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082. 2023-04-19 07:56:51,627 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6189702b8073b2778b13e3b2b656c082: 2023-04-19 07:56:51,627 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681890821595.6189702b8073b2778b13e3b2b656c082. 2023-04-19 07:56:51,628 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a96990349c624212b55fb1d571d42f5e 2023-04-19 07:56:51,805 DEBUG [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-19 07:56:52,006 DEBUG [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-19 07:56:52,033 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/.tmp/table/4f9e3a5963b34ee68893b70b98c0e1f0 2023-04-19 07:56:52,036 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4f9e3a5963b34ee68893b70b98c0e1f0 2023-04-19 07:56:52,036 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/.tmp/info/7619c3a991d044258f65252693cb030d as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/info/7619c3a991d044258f65252693cb030d 2023-04-19 07:56:52,039 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7619c3a991d044258f65252693cb030d 2023-04-19 07:56:52,039 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/info/7619c3a991d044258f65252693cb030d, entries=22, sequenceid=93, filesize=7.1 K 2023-04-19 07:56:52,041 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/.tmp/rep_barrier/a96990349c624212b55fb1d571d42f5e as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/rep_barrier/a96990349c624212b55fb1d571d42f5e 2023-04-19 07:56:52,044 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a96990349c624212b55fb1d571d42f5e 2023-04-19 07:56:52,044 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/rep_barrier/a96990349c624212b55fb1d571d42f5e, entries=6, sequenceid=93, filesize=5.5 K 2023-04-19 07:56:52,044 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/.tmp/table/4f9e3a5963b34ee68893b70b98c0e1f0 as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/table/4f9e3a5963b34ee68893b70b98c0e1f0 2023-04-19 07:56:52,045 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-19 07:56:52,045 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-19 07:56:52,047 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4f9e3a5963b34ee68893b70b98c0e1f0 2023-04-19 07:56:52,047 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/table/4f9e3a5963b34ee68893b70b98c0e1f0, entries=9, sequenceid=93, filesize=5.5 K 2023-04-19 07:56:52,048 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 845ms, sequenceid=93, compaction requested=false 2023-04-19 07:56:52,052 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-19 07:56:52,052 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-19 07:56:52,053 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-19 07:56:52,053 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-19 07:56:52,053 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-19 07:56:52,206 INFO [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42359,1681890819191; all regions closed. 2023-04-19 07:56:52,214 DEBUG [RS:0;jenkins-hbase4:42359] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/oldWALs 2023-04-19 07:56:52,214 INFO [RS:0;jenkins-hbase4:42359] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42359%2C1681890819191.meta:.meta(num 1681890821296) 2023-04-19 07:56:52,218 DEBUG [RS:0;jenkins-hbase4:42359] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/oldWALs 2023-04-19 07:56:52,218 INFO [RS:0;jenkins-hbase4:42359] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42359%2C1681890819191:(num 1681890821210) 2023-04-19 07:56:52,218 DEBUG [RS:0;jenkins-hbase4:42359] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:52,218 INFO [RS:0;jenkins-hbase4:42359] regionserver.LeaseManager(133): Closed leases 2023-04-19 07:56:52,218 INFO [RS:0;jenkins-hbase4:42359] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-19 07:56:52,219 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-19 07:56:52,220 INFO [RS:0;jenkins-hbase4:42359] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42359 2023-04-19 07:56:52,225 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42359,1681890819191 2023-04-19 07:56:52,225 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-19 07:56:52,225 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-19 07:56:52,226 ERROR [Listener at localhost/42517-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7e1d4b86 rejected from java.util.concurrent.ThreadPoolExecutor@7a22c09d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2302] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-19 07:56:52,226 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42359,1681890819191] 2023-04-19 07:56:52,227 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42359,1681890819191; numProcessing=1 2023-04-19 07:56:52,228 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42359,1681890819191 already deleted, retry=false 2023-04-19 07:56:52,228 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,42359,1681890819191 expired; onlineServers=0 2023-04-19 07:56:52,228 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43905,1681890817995' ***** 2023-04-19 07:56:52,228 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-19 07:56:52,228 DEBUG [M:0;jenkins-hbase4:43905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@342e66f4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-19 07:56:52,228 INFO [M:0;jenkins-hbase4:43905] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43905,1681890817995 2023-04-19 07:56:52,228 INFO [M:0;jenkins-hbase4:43905] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43905,1681890817995; all regions closed. 2023-04-19 07:56:52,229 DEBUG [M:0;jenkins-hbase4:43905] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-19 07:56:52,229 DEBUG [M:0;jenkins-hbase4:43905] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-19 07:56:52,229 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-19 07:56:52,229 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681890820795] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681890820795,5,FailOnTimeoutGroup] 2023-04-19 07:56:52,229 DEBUG [M:0;jenkins-hbase4:43905] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-19 07:56:52,229 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681890820796] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681890820796,5,FailOnTimeoutGroup] 2023-04-19 07:56:52,229 INFO [M:0;jenkins-hbase4:43905] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-19 07:56:52,229 INFO [M:0;jenkins-hbase4:43905] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-19 07:56:52,229 INFO [M:0;jenkins-hbase4:43905] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-19 07:56:52,230 DEBUG [M:0;jenkins-hbase4:43905] master.HMaster(1512): Stopping service threads 2023-04-19 07:56:52,230 INFO [M:0;jenkins-hbase4:43905] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-19 07:56:52,230 ERROR [M:0;jenkins-hbase4:43905] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (1898789116) connection to localhost/127.0.0.1:36491 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-19 07:56:52,230 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-19 07:56:52,230 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-19 07:56:52,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-19 07:56:52,231 INFO [M:0;jenkins-hbase4:43905] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-19 07:56:52,231 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-19 07:56:52,231 DEBUG [M:0;jenkins-hbase4:43905] zookeeper.ZKUtil(398): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-19 07:56:52,231 WARN [M:0;jenkins-hbase4:43905] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-19 07:56:52,231 INFO [M:0;jenkins-hbase4:43905] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-19 07:56:52,231 INFO [M:0;jenkins-hbase4:43905] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-19 07:56:52,232 DEBUG [M:0;jenkins-hbase4:43905] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-19 07:56:52,232 INFO [M:0;jenkins-hbase4:43905] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 07:56:52,232 DEBUG [M:0;jenkins-hbase4:43905] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 07:56:52,232 DEBUG [M:0;jenkins-hbase4:43905] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-19 07:56:52,232 DEBUG [M:0;jenkins-hbase4:43905] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 07:56:52,232 INFO [M:0;jenkins-hbase4:43905] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.99 KB heapSize=393.72 KB 2023-04-19 07:56:52,328 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-19 07:56:52,328 INFO [RS:0;jenkins-hbase4:42359] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42359,1681890819191; zookeeper connection closed. 2023-04-19 07:56:52,328 ERROR [Listener at localhost/42517-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1a800077 rejected from java.util.concurrent.ThreadPoolExecutor@7a22c09d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2302] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-19 07:56:52,329 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): regionserver:42359-0x1019b691e330001, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-19 07:56:52,329 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4b812443] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4b812443 2023-04-19 07:56:52,329 ERROR [Listener at localhost/42517-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@621f26a5 rejected from java.util.concurrent.ThreadPoolExecutor@7a22c09d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2302] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-19 07:56:52,330 INFO [Listener at localhost/42517] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-19 07:56:52,644 INFO [M:0;jenkins-hbase4:43905] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.99 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d5d7257b67be4b689ae170cabd00ab2f 2023-04-19 07:56:52,647 INFO [M:0;jenkins-hbase4:43905] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d5d7257b67be4b689ae170cabd00ab2f 2023-04-19 07:56:52,647 DEBUG [M:0;jenkins-hbase4:43905] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d5d7257b67be4b689ae170cabd00ab2f as hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d5d7257b67be4b689ae170cabd00ab2f 2023-04-19 07:56:52,650 INFO [M:0;jenkins-hbase4:43905] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d5d7257b67be4b689ae170cabd00ab2f 2023-04-19 07:56:52,650 INFO [M:0;jenkins-hbase4:43905] regionserver.HStore(1080): Added hdfs://localhost:36491/user/jenkins/test-data/0c4cb078-1afd-3fdb-ef60-2af7d7e2ef29/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d5d7257b67be4b689ae170cabd00ab2f, entries=65, sequenceid=578, filesize=24.7 K 2023-04-19 07:56:52,651 INFO [M:0;jenkins-hbase4:43905] regionserver.HRegion(2948): Finished flush of dataSize ~344.99 KB/353273, heapSize ~393.70 KB/403152, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 419ms, sequenceid=578, compaction requested=false 2023-04-19 07:56:52,651 INFO [M:0;jenkins-hbase4:43905] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-19 07:56:52,651 DEBUG [M:0;jenkins-hbase4:43905] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-19 07:56:52,654 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-19 07:56:52,654 INFO [M:0;jenkins-hbase4:43905] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-19 07:56:52,655 INFO [M:0;jenkins-hbase4:43905] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43905 2023-04-19 07:56:52,657 DEBUG [M:0;jenkins-hbase4:43905] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,43905,1681890817995 already deleted, retry=false 2023-04-19 07:56:52,758 INFO [M:0;jenkins-hbase4:43905] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43905,1681890817995; zookeeper connection closed. 2023-04-19 07:56:52,758 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-19 07:56:52,758 ERROR [Listener at localhost/42517-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6341d136 rejected from java.util.concurrent.ThreadPoolExecutor@2d1a4101[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3850] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-19 07:56:52,759 DEBUG [Listener at localhost/42517-EventThread] zookeeper.ZKWatcher(600): master:43905-0x1019b691e330000, quorum=127.0.0.1:58087, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-19 07:56:52,759 ERROR [Listener at localhost/42517-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2456b155 rejected from java.util.concurrent.ThreadPoolExecutor@2d1a4101[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3850] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-19 07:56:52,761 WARN [Listener at localhost/42517] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-19 07:56:52,766 INFO [Listener at localhost/42517] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-19 07:56:52,875 WARN [BP-754958237-172.31.14.131-1681890814670 heartbeating to localhost/127.0.0.1:36491] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-19 07:56:52,875 WARN [BP-754958237-172.31.14.131-1681890814670 heartbeating to localhost/127.0.0.1:36491] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-754958237-172.31.14.131-1681890814670 (Datanode Uuid a4cdd384-e065-4e1c-a4f7-c8d7ed63f27c) service to localhost/127.0.0.1:36491 2023-04-19 07:56:52,877 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/cluster_d18a6997-17eb-f6df-6ce9-376c883e1a31/dfs/data/data1/current/BP-754958237-172.31.14.131-1681890814670] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-19 07:56:52,878 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/626d1d37-2493-05d4-5c1e-b1f4d1e8dec0/cluster_d18a6997-17eb-f6df-6ce9-376c883e1a31/dfs/data/data2/current/BP-754958237-172.31.14.131-1681890814670] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-19 07:56:52,915 INFO [Listener at localhost/42517] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-19 07:56:53,093 INFO [Listener at localhost/42517] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-19 07:56:53,276 INFO [Listener at localhost/42517] hbase.HBaseTestingUtility(1293): Minicluster is down